You can run your dataset questions against your agent (not just an LLM) by defining a custom experiment task that invokes your agent for each input. In Phoenix, you upload your dataset, define your agent as the task function, and then run an experiment using `run_experiment`, passing your dataset, the agent task, and your evaluator(s). This will execute your agent for each example and apply the evaluator(s) to the outputs. You do not need to use a sandbox unless you want to author and run custom code evaluators directly in the Phoenix UI; sandboxes are for running Python/TypeScript evaluators server-side, not for running your agent logic itself. See <https://arize.com/docs/phoenix/cookbook/datasets-and-experiments/experiment-with-a-customer-support-agent#create-and-run-an-experiment|Experiment with a Customer Support Agent> and <https://arize.com/docs/phoenix/datasets-and-experiments/how-to-experiments/how-to-dataset-evaluators|How-to: Dataset Evaluators>.
For multi-step evaluation (e.g., evaluating each step of a multi-step agent), Phoenix supports evaluating intermediate steps by capturing traces or by structuring your experiment so that each step's output is available for evaluation. You can log evaluations for each span (step) using `SpanEvaluation`, or design your experiment to return intermediate results for evaluation. However, the UI and built-in experiment runner primarily focus on evaluating the final output; multi-step (span-level) evaluation is possible but may require manual instrumentation or custom code. See <https://arize.com/docs/phoenix/cookbook/evaluation/evaluate-an-agent|Evaluate a Talk-to-Your-Data Agent> and <https://arize-ai.slack.com/archives/C04R3GXC8HK/p1716404752.254739|Slack discussion on span evaluations>.
Would you like a step-by-step code example for setting up such an experiment?
Sources:
- <https://arize.com/docs/phoenix/cookbook/datasets-and-experiments/experiment-with-a-customer-support-agent#create-and-run-an-experiment|https://arize.com/docs/phoenix/cookbook/datasets-and-experiments/experiment-with-a-customer-support-agent#create-and-run-an-experiment>
- <https://arize.com/docs/phoenix/datasets-and-experiments/how-to-experiments/how-to-dataset-evaluators|https://arize.com/docs/phoenix/datasets-and-experiments/how-to-experiments/how-to-dataset-evaluators>
- <https://arize.com/docs/phoenix/cookbook/evaluation/evaluate-an-agent|https://arize.com/docs/phoenix/cookbook/evaluation/evaluate-an-agent>
- <https://arize-ai.slack.com/archives/C04R3GXC8HK/p1716404752.254739|https://arize-ai.slack.com/archives/C04R3GXC8HK/p1716404752.254739>
