Running Evals with a Custom Inference Server as Judge

Hi, would it be possible to run evals using my custom inference server as judge? without using openai models?

· Sorted by Oldest

Mikyo
·
Definitely possible - if you are using things like vLLM you can use OpenAI SDKs or use LiteLLM. Can't say we will be much help past that since we've mainly tested things via things like ollama.
Keerthi
·
Thankyou Mikyo is there any sample notebook for eval with openai sdk ? i have my openai-spec compatible inference server. So i want to use that url as the openai endpoint and use that for eval.
Mikyo
·
Most of our notebooks are openai based but this one uses ollama + litellm. I'm sure you could do something similar https://colab.research.google.com/github/Arize-ai/phoenix/blob/main/tutorials/evals/local_llm.ipynb
1

Mikyo
·
Definitely possible - if you are using things like vLLM you can use OpenAI SDKs or use LiteLLM. Can't say we will be much help past that since we've mainly tested things via things like ollama.
Keerthi
·
Thankyou Mikyo is there any sample notebook for eval with openai sdk ? i have my openai-spec compatible inference server. So i want to use that url as the openai endpoint and use that for eval.
Mikyo
·
Most of our notebooks are openai based but this one uses ollama + litellm. I'm sure you could do something similar https://colab.research.google.com/github/Arize-ai/phoenix/blob/main/tutorials/evals/local_llm.ipynb
1