Arize AI Community Icon

Home
Events

Arize AX Releases
Arize AX Support
Arize News
Arize Observe
Discussions
Introductions
Phoenix Support

Powered by Tightknit

Phoenix Support

Setting Up Custom Evaluators for Prompt Output Comparison | Arize AI Community

Phoenix Support

Setting Up Custom Evaluators for Prompt Output Comparison

·Mar 19, 2025 09:09 PM

HI! I want to set up some custom evaluators to compare the outputs of different prompt versions. How can set up evaluators in the the UI itself? I'm using golden datasets for this. Thanks!

4 comments

· Sorted by Oldest

RunLLM
APP
·
To set up custom evaluators in Phoenix, you can use the Prompt Playground to test and compare different prompt versions. The Playground allows you to run prompts over datasets and evaluate their performance. You can create custom evaluators by writing Python functions that return a boolean or numeric value, which will be recorded as the evaluation score. These evaluators can be used to assess the outputs of different prompt versions.For more detailed guidance, you can refer to the Phoenix documentation on using evaluators and testing prompts in the Playground. Would you like more detail on setting up these evaluators or using the Playground?Sources: - Testing a Prompt in the Playground - Phoenix documentation on using evaluators
💡 Hint: Mention RunLLM in the thread for followups.
Mikyo
·
Hey Stuti P. server side evaluators are on the roadmap but not shipped. John G. has some talks that can show you how you can just run evaluators in a notebook to update your playground experiments in the meantime.
Rohith P.
·
·
Hello Mikyo Is this feature to add evaluators on server side shipped?
Mikyo
·
·
We're currently working on them. the roadmap should be the best place to check on the team's status.