Custom Metrics for Document Set/Retriever in Current System

·May 09, 2024 03:52 AM

Is there anyway to bring my own "document set/retriever document" metrics? As in - lets say I have a "% of docs from a certain domain" - etc. Obviously I can measure this myself when I pull the traces down, but if I want to push it back up and associate it with the Documents bar I dont think this is currently supported/exposed right?

3 comments

· Sorted by Oldest

Mikyo
·
You could definitely send an eval to the retriever span. It won’t show on the same place but we can definitely make it more prominent!
👍1
Trantor D.
·
Silly question but how do I log something to the retriever node specifically? .log_evaluations(SpanEvaluations right?
Mikyo
·
Yup! the data just has to have an index on context.span_id, you can associate a label score and or an explanation

Mikyo
·
You could definitely send an eval to the retriever span. It won’t show on the same place but we can definitely make it more prominent!
👍1
Trantor D.
·
Silly question but how do I log something to the retriever node specifically? .log_evaluations(SpanEvaluations right?
Mikyo
·
Yup! the data just has to have an index on context.span_id, you can associate a label score and or an explanation