Enhancing Evaluator Input with Full Task_Run Data

Igor B. · 2025-08-11T10:37:18.325Z

Hi, is there a way evaluator can get as input not only eval(output: str, expected: Dict[str, Any], metadata) but full task_run, like {'dataset_example_id': 'R1wbGU6MQ==', 'output': 'some_output', 'repetition_number': 1, 'start_time': '2025-08-10T20:19:13.498761+00:00', 'end_time': '2025-08-10T20:19:13.498644+00:00', 'trace_id': 'f24c282d3b3ebdfad50b3574476e127f', 'error': None, 'id': 'RXhwZXJpFJ1bjoz', 'experiment_id': 'RXhwZXJpbWVuDoz'}

Enhancing Evaluator Input with Full Task_Run Data | Arize AI Community

4 comments

· Sorted by Oldest

RunLLM
APP
·
No, Phoenix evaluators do not support receiving the full task run dictionary as a direct argument. Evaluator functions can only access specific fields—such as input, output, expected, reference, and metadata—by naming them as parameters; the full experiment run object is not passed in or available as a single argument in the evaluator signature. This is confirmed in the documentation and code, which specify the allowed parameter names and binding logic for evaluators, but do not mention or expose the full run object as an input parameter (<https://github.com/Arize-ai/phoenix/blob/main/packages/phoenix-client/src/phoenix/client/resources/experiments/__init__.py|source>, <https://github.com/Arize-ai/phoenix/blob/main/docs/datasets-and-experiments/how-to-experiments/using-evaluators.md|source>, <https://github.com/Arize-ai/phoenix/blob/main/src/phoenix/experiments/functions.py|source>). If you need additional fields from the run, you must add them to the output or metadata of your task so they can be accessed by name in the evaluator. Would you like more detail or a code example? Sources: - <https://github.com/Arize-ai/phoenix/blob/main/packages/phoenix-client/src/phoenix/client/resources/experiments/__init__.py|phoenix/client/resources/experiments/__init__.py> - <https://github.com/Arize-ai/phoenix/blob/main/docs/datasets-and-experiments/how-to-experiments/using-evaluators.md|Using Evaluators> - <https://github.com/Arize-ai/phoenix/blob/main/src/phoenix/experiments/functions.py|https://github.com/Arize-ai/phoenix/blob/main/src/phoenix/experiments/functions.py>
💡 Hint: Mention RunLLM in the thread for followups.
Igor B.
·
I would like to build custom evaluator which depends on inner spans of task_run, not only output
Igor B.
·
how can I add trace_id from task_run to metadata that could be accessible in evaluator?
Xander S.
·
Hey Igor B., makes a ton of sense. I don't think we currently support that feature, but I've filed an issue here. Please feel free to add color to the ticket! cc Dustin N. Elizabeth H.
🙏2

RunLLM
APP
·
No, Phoenix evaluators do not support receiving the full task run dictionary as a direct argument. Evaluator functions can only access specific fields—such as input, output, expected, reference, and metadata—by naming them as parameters; the full experiment run object is not passed in or available as a single argument in the evaluator signature. This is confirmed in the documentation and code, which specify the allowed parameter names and binding logic for evaluators, but do not mention or expose the full run object as an input parameter (<https://github.com/Arize-ai/phoenix/blob/main/packages/phoenix-client/src/phoenix/client/resources/experiments/__init__.py|source>, <https://github.com/Arize-ai/phoenix/blob/main/docs/datasets-and-experiments/how-to-experiments/using-evaluators.md|source>, <https://github.com/Arize-ai/phoenix/blob/main/src/phoenix/experiments/functions.py|source>). If you need additional fields from the run, you must add them to the output or metadata of your task so they can be accessed by name in the evaluator. Would you like more detail or a code example? Sources: - <https://github.com/Arize-ai/phoenix/blob/main/packages/phoenix-client/src/phoenix/client/resources/experiments/__init__.py|phoenix/client/resources/experiments/__init__.py> - <https://github.com/Arize-ai/phoenix/blob/main/docs/datasets-and-experiments/how-to-experiments/using-evaluators.md|Using Evaluators> - <https://github.com/Arize-ai/phoenix/blob/main/src/phoenix/experiments/functions.py|https://github.com/Arize-ai/phoenix/blob/main/src/phoenix/experiments/functions.py>
💡 Hint: Mention RunLLM in the thread for followups.
Igor B.
·
I would like to build custom evaluator which depends on inner spans of task_run, not only output
Igor B.
·
how can I add trace_id from task_run to metadata that could be accessible in evaluator?
Xander S.
·
Hey Igor B., makes a ton of sense. I don't think we currently support that feature, but I've filed an issue here. Please feel free to add color to the ticket! cc Dustin N. Elizabeth H.
🙏2