hi! I'm currently exploring RAG observability metrics, specifically MAP (Mean Average Precision) and MRR (Mean Reciprocal Rank). Is there anyway I can leverage the repo below? https://github.com/Arize-ai/phoenix/blob/main/tutorials/evals/evaluate_rag.ipynb
Hey Namira S.! Thanks so much for the context. Super interesting use-case and would love to unblock you. I guess I'd start with the CSV as a dataset: Yes this will be supported but is coming via a new feature called datasets (https://github.com/Arize-ai/phoenix/issues/2017) - you will then be able to run experiments with these datasets to test changes to your application or retrieval. I think this might be ultimately what you are looking for? If so would love to speak to you more and get some early feedback. Would you be open to this? In the meantime however the best way to evaluate RAG with phoenix is to first trace your application. We currently don't have an example with LangGraph specifically but we do have quite a few langchain ones (https://docs.arize.com/phoenix/notebooks). You can then transform traces -> dataframes via queries (https://docs.arize.com/phoenix/tracing/how-to-tracing/extract-data-from-spans) and you can ultimately label the spans themselves by just providing the metrics you calculate (https://docs.arize.com/phoenix/tracing/how-to-tracing/llm-evaluations). I recently gave a talk on this exact flow that might help (https://www.youtube.com/watch?v=vIKW-8YWCSg) Hope that makes sense. Thanks again for sharing your project!
