Andrew S.

Right now it’s simply evaluating precision and recall for retrieval. So false negative/false positive with an optional note is all I really need at the moment. https://www.loom.com/share/b4c322552955434a9059c17e22438e30

exactly! that would be amazing. It would dramatically improve the user experience for non-technical domain experts. My goal is to enable them to work comfortably and effectively entirely within Phoenix without needing developers in the loop. As it stands, there is little they can do in Phoenix without relying on developers to run scripts (experiments and processing traces to populate datasets). I want them to be able to use our application then easily find the relevant traces within Phoenix, play with them in the prompt playground (i.e., running ad-hoc LLM judge prompts against traces), annotate traces, and add examples to datasets. Right now this workflow is high friction. thanks for considering 🙏

RunLLM this is not great because I cannot simply add traces to a dataset and then immediately use them in the prompt playground

RunLLM my dataset does include a reference column but the only possible variable to use is input . is this a bug? are there any issues or discussion about this?

RunLLM in my case, I want to use the prompt playground to run a prompt over a dataset of traces of the retriever type. my retrieved documents are reference outputs. therefor, I want the LLM to see these outputs and judge the retrieved results. that’s why I need to be able to reference the reference outputs (not just inputs) in my prompt. does that make sense? or am I misusing phoenix?

RunLLM i see this error: Dataset is missing input for variable: reference. Possible inputs are: input

yes. are there plans to support this in the UI RunLLM

yes. are there plans to support this in the UI?