Using Self-Hosted LLMs in Phoenix Prompt Playground and Trace Collection

Prerit S. · 2025-03-06T22:31:31.504Z

Hello, I have a question regarding Prompt Playground in Phoenix. If we have self-hosted LLMs (that are served in a OpenAI compatible manner), how can we use them in Prompt Playground? Also, Can we use any auto-instrumentation to collect traces in self-hosted Phoenix if we are using self hosted LLM deployed using vLLM and KServe?

RunLLM
APP
·
It seems likely that you can use self-hosted LLMs in the Phoenix Prompt Playground if they are OpenAI compatible. You can configure the base URL for the OpenAI client to point to your self-hosted LLM endpoint. This is done by setting the `OPENAI_BASE_URL` environment variable to your LLM's endpoint, as mentioned in the <https://github.com/Arize-ai/phoenix/blob/main/docs/prompt-engineering/how-to-prompts/configure-ai-providers.md|Prompt Playground documentation>. Regarding auto-instrumentation for collecting traces in a self-hosted Phoenix setup, it seems that Phoenix supports tracing for LLM applications using OpenTelemetry. You can use the `LiteLLMInstrumentor` for auto-instrumentation if your LLM is compatible with the OpenAI API format. This is supported by the <https://github.com/Arize-ai/phoenix/blob/main/docs/tracing/integrations-tracing/litellm.md|LiteLLM integration> in Phoenix. For vLLM and KServe, you might need to ensure that your setup is compatible with OpenTelemetry to send traces to Phoenix. Would you like more detail on any specific part? Sources: - <https://github.com/Arize-ai/phoenix/blob/main/docs/prompt-engineering/how-to-prompts/configure-ai-providers.md|Configure AI Providers> - <https://github.com/Arize-ai/phoenix/blob/main/docs/tracing/integrations-tracing/litellm.md|LiteLLM integration>
💡 Hint: Mention RunLLM in the thread for followups.
Roger Y.
·
yes, you can try setting the OPENAI_BASE_URL environment variable, or specify it in the UI
Roger Y.
·
phoenix already collects the playground traces automatically, so you try take a look at those first
Prerit S.
·
Where to set env variables? In Python code?
Roger Y.
·
do you have config fils for your self hosted phoenix?
Prerit S.
·
yes we do...
Roger Y.
·
you can add it alongside others already in there, like the ones for the database, etc
Prerit S.
·
but we may use more than one self-hosted models in which case base url might be different for each model
Roger Y.
·
oh i see. that would be a new feature
Prerit S.
·
but I guess, in playground, base URL field we can use where we can plug in different values
Prerit S.
·
likewise in the code too..
Roger Y.
·
yes
Prerit S.
·
ok..got it..thank you
Roger Y.
·
np!
Prerit S.
·
will give it a try

Using Self-Hosted LLMs in Phoenix Prompt Playground and Trace Collection

16 comments

Using Self-Hosted LLMs in Phoenix Prompt Playground and Trace Collection

16 comments