Request for Assistance with Arize AI Phoenix Tool

Sangram · 2024-01-10T04:50:47.233Z

Hii Xander & Arize Ai Team! Sangram this side our organization has built an RAG system and we want to evaluate it using Arize ai phoniex. but there are too many problems we are facing while using your phoniex. As we are new to this tool we are not able to understand it properly, or how to use it. it would be really helpful if you or anyone from your team or company could help us with it?

Mikyo
·
Hi Sangram 👋 can you tell us a little bit more about your deployment and the problems you are facing? Feel free to post here or start via an issue on our GitHub!
Sangram
·
Hi, Mikyo actually we have built an LLM-based RAG model. and we want to evaluate its performance. so we tried different frameworks such as Ragas, Vectara, etc. we want to evaluate metrics such as hallucinations, context_precision, answer_relevancy, faithfulness, etc. so we have test cases data which have questions, model_response & context. so by using this, we have to evaluate using your Phoenix framework. but we are not able to understand how it works and how should we ingest our data?? so if we can connect virtually it will be helpful for us???
🙌1
Mikyo
·
Hi Sangram ! Yes we’d be happy to meet your team. We internally use our evals to optimize our evals so we would have some ideas on how to use them best. But also happy to answer any questions any questions or confusion that might exist with regards to our evals. https://docs.arize.com/phoenix/llm-evals/llm-evals
✨1
Sangram
·
Thanks Mikyo! i have confusion, in the following colab file of hallucination, you have used lablelled data. but we want to check our test cases are hallucinated or not? so how can we do that? we have question, answer & context from our RAG. on basis of these
Sangram
·
3 column we want to evaluate our RAG? so how can we do that?
Mikyo
·
Yup that’s perfect! We just take those 3 columns in a Pandas dataframe as input. It just needs to be those names: input, output, context. With that you can run our evaluations using an eval model, say GPT4 Turbo
Mikyo
·
Hopefully that helps… let us know if you get stuck. You don’t need labelled data. That’s just to show how we’ve benchmarked our evals. The evaluation itself requires no ground truth labels
Sangram
·
Thanks Mikyo just one more question, actually we are using azure Openai api?? so not able to run this code on azure openai??
Sangram
·
can you please guide me, how can we use Azure Openai instead of GPT4?
Mikyo
·
Yes Sangram you can just configure Azure OpenAI using your deployment https://docs.arize.com/phoenix/api/evaluation-models#azure-openai
Mikyo
·
Something like model = OpenAIModel( model_name="gpt-4-32k", azure_endpoint="https://YOUR_SUBDOMAIN.openai.azure.com/", api_version="2023-03-15-preview" )
Sangram
·
Hey Mikyo I tried as you suggested above but it gave errors like this:
Sangram
·
and where to pass azure openai API?
Sangram
·
we are trying for a long time but have not been able to crack it ☹️
Mikyo
·
I may not be able to troubleshoot exactly what's going wrong but double check that you have set your OPENAI_API_KEY correctly and you are matching your deployment?

68 comments