Questions on Using Phoenix for RAG System Benchmarking

·Sep 02, 2024 04:40 PM

Hi all, After starting to use phoenix (specifically run_experiment() ) to benchmark our agentic RAG system on a golden dataset, a few questions came up:

1.
We are storing input parameters associated with an experiment eg. model id, chunking strategy, system prompt etc. in the Phoenix experiment metadata. Is this what you would recommend, or is there a better place to put it? Also, I'm curious if it is possible to do a diff on the metadata to track changes between experiments?
2.
Is there any functionality around labelling ie. to be able to run an experiment over a subset of examples in a dataset, or subset of evaluators for certain examples? Perhaps the best way of handling this would be the metadata, or the dataset we upload?
3.
What is the best way to get run_experiment results into Arize? I have been trying to piece together a dataframe with label, score, explanation, and span_id columns from the output of run_experiment (the RanExperiment class), but I currently can't find the span_id, and it is getting a bit complex. I wondered if there is a straightforward way to do it? It would be nice to be able to see what Arize offers, in terms of sorting experiments in order of decreasing evaluation metric score, applying other filtering to results, and graphing results over time and things, which I don't believe is possible in Phoenix? I have Phoenix experiment data persisted in a SQLite database in case that simplifies getting data into Arize?
4.
For RAG evaluations, I'm displaying a JSON with the question, answer, and source documents in the Phoenix UI as the output for each example in an experiment, as it seems that is necessary to feed all the information into the evaluators (given evaluator input and task output displayed by the UI are the same thing). Is that best practice? It is slightly harder to digest by eye when looking over the results, but not a huge deal.

Thanks!

Phoenix Support

Questions on Using Phoenix for RAG System Benchmarking

Simon W.

·Sep 02, 2024 04:40 PM

Hi all, After starting to use phoenix (specifically run_experiment() ) to benchmark our agentic RAG system on a golden dataset, a few questions came up:

1.
We are storing input parameters associated with an experiment eg. model id, chunking strategy, system prompt etc. in the Phoenix experiment metadata. Is this what you would recommend, or is there a better place to put it? Also, I'm curious if it is possible to do a diff on the metadata to track changes between experiments?
2.
Is there any functionality around labelling ie. to be able to run an experiment over a subset of examples in a dataset, or subset of evaluators for certain examples? Perhaps the best way of handling this would be the metadata, or the dataset we upload?
3.
What is the best way to get run_experiment results into Arize? I have been trying to piece together a dataframe with label, score, explanation, and span_id columns from the output of run_experiment (the RanExperiment class), but I currently can't find the span_id, and it is getting a bit complex. I wondered if there is a straightforward way to do it? It would be nice to be able to see what Arize offers, in terms of sorting experiments in order of decreasing evaluation metric score, applying other filtering to results, and graphing results over time and things, which I don't believe is possible in Phoenix? I have Phoenix experiment data persisted in a SQLite database in case that simplifies getting data into Arize?
4.
For RAG evaluations, I'm displaying a JSON with the question, answer, and source documents in the Phoenix UI as the output for each example in an experiment, as it seems that is necessary to feed all the information into the evaluators (given evaluator input and task output displayed by the UI are the same thing). Is that best practice? It is slightly harder to digest by eye when looking over the results, but not a huge deal.

Thanks!

def skipping_evaluator(output, expected, metadata): if output is None: # skipped in the task run return "skipped" # the evaluation label will be "skipped" if metadata.get("skip_evaluation", False): return "skipped" # skip the evaluation on metadata return original_evaluation(output, expected)

from phoenix.experiments.types import EvaluationResult skipped_result = EvaluationResult(score=0.0, label="skipped-due-to-metadata-flag") def skipping_evaluator(output, expected, metadata): if metadata.get("skip_evaluation", False): return skipped_result # skip the evaluation on metadata return EvaluationResult(score=1, label="not=skipped") # or any arbitrary code to compute the score

import phoenix import pandas as pd from phoenix.experiments import run_experiment import pandas as pd import random phoenix.launch_app() phoenix_client = phoenix.Client() questions = [ "What is the capital of France?", "Who wrote 'Romeo and Juliet'?", "What is the largest planet in our solar system?", "What year did World War II end?", "What is the chemical symbol for gold?" ] answers = [ "Paris", "William Shakespeare", "Jupiter", "1945", "Au" ] df = pd.DataFrame({ 'Questions': questions, 'Ground-Truth Answers': answers }) dataset = phoenix_client.upload_dataset( dataframe=df, dataset_name="test", input_keys=['Questions'], output_keys=['Ground-Truth Answers'], ) def task(input): return random.choice(["Sorry, I am unable to assist you with this request.", "non-refusal"]) def refusal_evaluator(input, output, expected): refusal_response = "Sorry, I am unable to assist you with this request." if output == refusal_response: print(f"refusal: \n Question: {input['Questions']}, \n Answer: {output}\n\n") return 1.0 else: print(f"non-refusal: \n Question: {input['Questions']}, \n Answer: {output}\n\n") return 0.0 run_experiment( dataset=dataset, task=task, evaluators=[refusal_evaluator], experiment_name="test_experiment", experiment_description="reproducing misaligned evaluators and examples", )

19 comments