Another question -- If I have a dataset with LLM calls but then I annotate the source trace. Can we get the metadata to automatically populate in the dataset?
Yes, Phoenix allows annotations from traces to be propagated to datasets. When you annotate a trace, the annotations are included in the span metadata under a new key called annotations. This metadata is then preserved when the trace data is used to create or update a dataset, allowing the annotations to be automatically included in the dataset's metadata. This feature ensures that any labels or scores assigned during the annotation process are carried over to the dataset for further analysis or experimentation.Would you like more detail on how to implement this? Sources:
- Annotating Traces
- Add span annotations to dataset example metadata
馃挕聽Hint: Mention RunLLM in the thread for followups.
Xander S. We're looking for a way to save the human annotations, which I would've expected to propagate, but doesn't. Kashyap C. I'll go ahead and create this ticket now.