Using a Self-Hosted LLM Endpoint in OpenAI Playground

Anton A. · 2025-01-29T17:09:17.750Z

Hey, is it possible to use a self-hosted Open AI compatible LLM endpoint in the playground? Can’t see any way to configure this. The only options listed are: Open AI, Azure Open AI, Anthropic and Google AI Studio.

Using a Self-Hosted LLM Endpoint in OpenAI Playground | Arize AI Community

RunLLM
APP
·
Yes, it is possible to use a self-hosted OpenAI-compatible LLM endpoint in the Phoenix playground. You can set the OPENAI_BASE_URL environment variable in your configuration, such as in a docker-compose.yml file, to point to your self-hosted endpoint. This allows the playground to interact with your custom LLM endpoint as if it were an OpenAI endpoint, provided it follows the OpenAI API specifications.
Would you like more detail on how to set this up?
Sources: - GitHub Issue #5836
💡 Hint: Mention RunLLM in the thread for followups.
Anton A.
·
The issue linked is asking a question… it doesn’t seem to be supported
Roger Y.
·
Sorry, it’s not supported yet. cc. Anthony P.
Anthony P.
·
Yeah this is definitely on our radar but is not yet implemented, definitely throw a thumbs up on the linked issue above and we can get it prioritized 🙏
Anton A.
·
Alright, thanks Tony! This would be great — major blocker for us to start using Arize internally since we self host models
Anton A.
·
Is this something that would be trivial to fix you think? I could submit an MR if you don’t think the lift is that big
Anton A.
·
Seems like all that’s needed is just an extra config field to store OPENAI_BASE_URL in addition to OPENAI_API_KEY so could follow what’s done for api key
Anthony P.
·
I wouldn't call it trivial as it spans a few components of the backend and into the query layer of the frontend, but at the same time I would never discourage someone from hacking on the codebase 😅
Anton A.
·
Fair, I’ll pull the codebase and see what I can do 🙂
🙌1
Anthony P.
·
Let us know if you have trouble getting spun up, can always use more feedback on the development readme
👍1
Roger Y.
·
We’re starting to work on an enhancement for this on our end
Anton A.
·
Great, thanks for the update!
Roger Y.
·
in 7.10.0 you can now set the Base URL under OpenAI
Roger Y.
·
Please try it out and let us know if you have questions. Thanks for using Phoenix!
Anton A.
·
Awesome, that was so quick! I’ll try it out tomorrow 🙂

Using a Self-Hosted LLM Endpoint in OpenAI Playground

38 comments

Using a Self-Hosted LLM Endpoint in OpenAI Playground

38 comments