You changed a router prompt, swapped a model, or updated a tool, and the failure you were debugging disappeared. But did the change actually improve the agent across the inputs and workflows you care about?
A single-prompt test can't answer that. Agent experiments let you run a dataset through the full deployed agent, including routing, tool calls, and multi-step orchestration, so you can measure the effect of a change across the system.
In this session, we'll run that workflow live in Arize AX. We'll register a deployed agent as a remote agent, launch an experiment from a dataset, and inspect the traces behind each result, score runs with evaluators, and compare experiments side by side.
The goal? Once you find a failure, you should be able to test the fix across the full agent and catch regressions before you ship a change.
Who this is for
AI engineers and agent builders shipping LangGraph, CrewAI, OpenAI Agents SDK, Claude Agent SDK, or custom agents into production, plus PMs and technical leads responsible for agent quality and reliability.
This is especially relevant if you've ever fixed a failing trace and then wondered whether the change made the rest of the agent better or worse.
What you'll leave with
The three experiment types and when to use each: playground (one prompt), code (a Python function), and agent (the full deployed agent).
How to test routing, tool, and model changes across your agent.
How to register your agent as a remote agent so anyone in the space can run experiments, no code needed.
How to launch an experiment from a dataset using {{dataset.column}} templating.
How Arize runs every dataset row against your endpoint in parallel, one experiment run per row.
How traces link back automatically, so you see every LLM and tool call.
How to attach evaluators and compare runs to prove the whole agent improved, not just one output.
Format
30 minutes of presentation and live demo, 15 minutes of Q&A.
Level
Intermediate. Assumes familiarity with LLM agents, traces, and basic evaluation concepts. No prior Arize experience required.
