Integrating LiteLLM with Llama_index: Seeking MWE Example

·May 30, 2024 04:58 PM

I am trying to do an evaluation with Phoenix/Arize with Llama_index, but using an OpenSource model. LiteLLM is supposedly supported, but when I use it within Llama_index, I get errors such as field default_concurrency missing, and once I created such a field, the message reload-client missing. Needless to say, I must be doing something wrong or missing some important point. Can somebody please point me to, or provide me with, a MWE example using Llama_index and an open source model, with the use of an evaluator, for example, ” hallucination_evaluator = HallucinationEvaluator(model=eval_llm)” If I am posting in the wrong channel, I apologize. Gordon

10 comments

· Sorted by Oldest

Mikyo
·
Hey Gordon E. you are in the right channel 👋 So with LiteLLM, you don't have to use it in llama-index itself but it does need to be used as the eval_llm. Are you seeing the error during execution of evals? Sorry just trying to understand where the error is. Any chance you have a stack trace?
Gordon E.
·
Hi @mikyo! Yes, the error occurs when I call eval. But to make sure we are on the same page, I am attaching my code (short) and the error message (longer). Can you read the file (it is labelled binary but it is text). I realize that I can run outside Llama_index, but I thought that Phoenix was integrated into Llama_index. I thought that included evaluation. Thanks!
Mikyo
·
Hey Gordon E. so I know this is a bit confusing but we have our own model instances for evals that have concurrency optimizations. So when you pass it into evals it needs to be an instance of this model. https://docs.arize.com/phoenix/api/evaluation-models#litellmmodel
Mikyo
·
src: https://github.com/Arize-ai/phoenix/blob/main/packages/phoenix-evals/src/phoenix/evals/models/litellm.py
Mikyo
·
does that make sense? I can create a minimum working example if not
Gordon E.
·
A MWE would be appreciated. If it is possible to use a non-openAI model, that would be even better. Thanks.
Gordon E.
·
I was able to get relevance, correctness, and hallucinations, which is good. But I had to use GPT3.5 for this. I still want to be able to use an Open Source model.

Mikyo

Hey Gordon, was working on a MWE for ya. I have it working using llama3 using ollama (https://github.com/Arize-ai/phoenix/pull/3348)

import os

from phoenix.evals import LiteLLMModel

os.environ["OLLAMA_API_BASE"] = "http://localhost:11434"

model = LiteLLMModel(model="ollama/llama3")

Mikyo
·
Definitely check out our benchmarks on open-source models. Quality might vary a bit. https://docs.arize.com/phoenix/evaluation/how-to-evals/running-pre-tested-evals
Gordon E.
·
Thanks! I was able to perform an evaluation of data produced by Phi3 (Ollama) using GPT3.5. But I want to perform an evaluation with an open-source model, without the need for Internet. Cheers!
🙌1

Mikyo
·
Hey Gordon E. you are in the right channel 👋 So with LiteLLM, you don't have to use it in llama-index itself but it does need to be used as the eval_llm. Are you seeing the error during execution of evals? Sorry just trying to understand where the error is. Any chance you have a stack trace?
Gordon E.
·
Hi @mikyo! Yes, the error occurs when I call eval. But to make sure we are on the same page, I am attaching my code (short) and the error message (longer). Can you read the file (it is labelled binary but it is text). I realize that I can run outside Llama_index, but I thought that Phoenix was integrated into Llama_index. I thought that included evaluation. Thanks!
Mikyo
·
Hey Gordon E. so I know this is a bit confusing but we have our own model instances for evals that have concurrency optimizations. So when you pass it into evals it needs to be an instance of this model. https://docs.arize.com/phoenix/api/evaluation-models#litellmmodel
Mikyo
·
src: https://github.com/Arize-ai/phoenix/blob/main/packages/phoenix-evals/src/phoenix/evals/models/litellm.py
Mikyo
·
does that make sense? I can create a minimum working example if not
Gordon E.
·
A MWE would be appreciated. If it is possible to use a non-openAI model, that would be even better. Thanks.
Gordon E.
·
I was able to get relevance, correctness, and hallucinations, which is good. But I had to use GPT3.5 for this. I still want to be able to use an Open Source model.

Mikyo

Hey Gordon, was working on a MWE for ya. I have it working using llama3 using ollama (https://github.com/Arize-ai/phoenix/pull/3348)

import os

from phoenix.evals import LiteLLMModel

os.environ["OLLAMA_API_BASE"] = "http://localhost:11434"

model = LiteLLMModel(model="ollama/llama3")

Mikyo
·
Definitely check out our benchmarks on open-source models. Quality might vary a bit. https://docs.arize.com/phoenix/evaluation/how-to-evals/running-pre-tested-evals
Gordon E.
·
Thanks! I was able to perform an evaluation of data produced by Phi3 (Ollama) using GPT3.5. But I want to perform an evaluation with an open-source model, without the need for Internet. Cheers!
🙌1