Arize AI Community Icon

Home
Events

Arize AX Releases
Arize AX Support
Arize News
Arize Observe
Discussions
Introductions
Phoenix Support

Powered by Tightknit

Evaluating Test Cases in Arize Phoenix: A Quick Guide | Arize AI Community

Evaluating Test Cases in Arize Phoenix: A Quick Guide

·Jan 16, 2025 10:58 PM

Hi I attended a talk on arize phoenix yesterday which looked very promising. I had a question on the testcases that I can evaluate with phoenix. Do I have to create the test cases myself or are they dynamically generated with LLM using Phoenix?

9 comments

· Sorted by Oldest

Xander S.
·
Hey Omar Q., both are possible. We are agnostic with respect to the origin of the data in datasets and provide helpers such as llm_generate to help you create synthetic data.
👍1
Xander S.
·
It's also possible to curate examples from spans you have collected from production data.
👍1
Omar Q.
·
Do u guys have any testsets published for common tasks like rag, hallucinations detection, etc? Xander S.
Xander S.
·
We do have a collection of datasets that we use to benchmark the effectiveness of our default LLM evaluators. We typically find that users want to tailor datasets to their particular use case/ application. John G. may have more to add on benchmarks as well.
👍1
Omar Q.
·
Cool thanks, super helpful
Omar Q.
·
It would be awesome if u guys could publish more
👍1
Xander S.
·
Definitely let us know what kind of content or talks you want to see! Might be worth checking out our blog and YouTube channel as well.
👍1
John G.
·
Omar Q. - Xander covered most of the details here! Just to understand your ideal situation - you're hoping we'd provide test cases for common evaluations that you could use off-the-shelf? Or give you the ability to generate test cases within the product for your specific case? We're doing some dataset creation work now, but it leans more towards the first option
👍1
Omar Q.
·
Both John G., testsets for the common evals and the ability to generate test cases dynamically based on my use case, so the ability to generate test cases for my agent, testing the tooling, response, etc.