Arize AI Community Icon

Home
Events

Arize AX Releases
Arize AX Support
Arize News
Arize Observe
Discussions
Introductions
Phoenix Support

Powered by Tightknit

Automatically Populate Metadata for LLM Dataset with Annotations | Arize AI Community

Phoenix Support

Phoenix Support

Automatically Populate Metadata for LLM Dataset with Annotations

·Feb 27, 2025 05:55 PM

Another question -- If I have a dataset with LLM calls but then I annotate the source trace. Can we get the metadata to automatically populate in the dataset?

7 comments

· Sorted by Oldest

RunLLM
APP
·
Yes, Phoenix allows annotations from traces to be propagated to datasets. When you annotate a trace, the annotations are included in the span metadata under a new key called annotations. This metadata is then preserved when the trace data is used to create or update a dataset, allowing the annotations to be automatically included in the dataset's metadata. This feature ensures that any labels or scores assigned during the annotation process are carried over to the dataset for further analysis or experimentation.Would you like more detail on how to implement this? Sources: - Annotating Traces - Add span annotations to dataset example metadata
💡 Hint: Mention RunLLM in the thread for followups.
Xander S.
·
Kashyap C. I believe this was added to Phoenix v7.10.0.
https://github.com/Arize-ai/phoenix/pull/6163
https://github.com/Arize-ai/phoenix/releases/tag/arize-phoenix-v7.10.0
👀1
Samantha Z.
·
Xander S. We're looking for a way to save the human annotations, which I would've expected to propagate, but doesn't. Kashyap C. I'll go ahead and create this ticket now.
Xander S.
·
Samantha Z. It sounds like you want the annotations on traces to propagate to the dataset examples?
Samantha Z.
·
No, I want manual feedback to show up in Feedback, and right now it doesn't?
Xander S.
·
Not sure I follow. Can you send some screenshots of the original annotation and where you would like it to show up?
Samantha Z.
·
So if we annotate somewhere down the chain, it shows up in feedback for just that span, but we can't see it from the top level or from the dashboard.