Using Phoenix Evals with Local Models Like Mistral and Falcon

AS

·Jan 19, 2024 10:22 PM

Hey All, need some directions on using phoenix evals on locally downloaded models like mistral/falcon etc

18 comments

· Sorted by Oldest

Xander S.
·
Hey AS, are you trying to run these models via HuggingFace?
AS
·
Hi Xander S., I have downloaded the llm on to my local system
👍1
Xander S.
·
Curious if you're using a library such as HuggingFace as an interface to the models, or if you're running the model code directly.
AS
·
i have downloaded the model file from huggingface model file
👍1
Xander S.
·
Sounds like you're trying to run the model on bare metal as opposed to via a third-party API?
AS
·
Right, so basically i want to use the falcon/mistral do hallucination evaluation instead of openai
Xander S.
·
Got it. At the moment, I don't think we have a HuggingFace adapter for these models to run Phoenix evals on bare metal, although I believe Jason has some experience using our LiteLLM adapter to run Mixtral on Anyscale Endpoints.
Xander S.
·
We began looking into building a HuggingFace adapter a while back and can re-open the issue.
Xander S.
·
One thing to note is that the throughput for evals would be greatly constrained if you're running on a single node/ GPU. We are able to use concurrency to compute large volumes of evals quickly with third-party APIs.
AS
·
Thanks for your inputs
👍1
Xander S.
·
Is your main motivation around using open-source models around data privacy/ not sending data to third-party APIs, or something else? If you are interested in using Mixtral or Falcon and are able to use a service like Anyscale Endpoints, it's straightforward to get set up.
Jason
·
I have got Huggingface to work through LLMlite
Xander S.
·
Oh nice, I did not know that.
Jason
·
I can dig it up
👍1
AS
·
Is your main motivation around using open-source models around data privacy/ not sending data to third-party APIs - Yes For that I need to download an open source llm and work it locally
👍2