Debugging Feedback Update Issues in Arize-Phoenix 8.6.1

James S. · 2025-02-28T13:28:12.147Z

Hi, I’m running experiments and log evaluation results to phoenix. Sometimes the feedback isn’t updated. How can I debug the problem? The code is here: https://gist.github.com/goofansu/d01464f95ead23695e8075c0020f4a61 arize-phoenix version: 8.6.1

Debugging Feedback Update Issues in Arize-Phoenix 8.6.1 | Arize AI Community

RunLLM
APP
·
To debug the issue of feedback not updating in Phoenix, consider the following steps: 1. *Check Environment Variables*: Ensure that the environment variables such as `PHOENIX_COLLECTOR_ENDPOINT` are correctly set to point to the Phoenix server. This is crucial for the communication between your application and Phoenix (<https://arize-ai.slack.com/archives/C04R3GXC8HK/p1722778288.923159|source>). 2. *Verify API Usage*: Make sure you are using the correct API endpoints and methods for logging evaluations. The `px.Client().log_evaluations()` method should be used correctly with the appropriate dataframes (<https://github.com/Arize-ai/phoenix/issues/6525|source>). 3. *Check for Errors*: Look at the logs of your application and the Phoenix server for any errors or warnings that might indicate why feedback is not being updated. This can provide insights into any issues with the data being sent or received (<https://arize-ai.slack.com/archives/C04R3GXC8HK/p1722778288.923159|source>). Would you like more detailed steps or specific code examples? Sources: - <https://arize-ai.slack.com/archives/C04R3GXC8HK/p1722778288.923159|https://arize-ai.slack.com/archives/C04R3GXC8HK/p1722778288.923159> - <https://github.com/Arize-ai/phoenix/issues/6525|https://github.com/Arize-ai/phoenix/issues/6525>
💡 Hint: Mention RunLLM in the thread for followups.
Dustin N.
·
Hi James S. sorry you're running into this, is there any more detail you can give about when the annotations don't seem to make it to the server?
Roger Y.
·
these evals are sent before the span is received by the server, so there’s a chance that they missed each other just long enough that the server has nowhere to put the evals
Roger Y.
·
also, these are evals of experiment “evaluator”
Roger Y.
·
so maybe there could be a better approach here since they are capturing the same info
Roger Y.
·
i guess the output of the “evaluator” could be surfaced better in the UI
Roger Y.
·
i think these “evaluator” spans shouldn’t need to be annotated (for a second time), since they’re mostly there for record keeping after running the evaluators. The output of the evaluator should have already been used to annotate the experiment runs themselves, which are the real subjects of evaluation.
Roger Y.
·
maybe we should surface the output of the evaluators in the output column of the table. that would show the same info as what the screenshot is attempting to do
James S.
·
Dustin N. No worry, maybe I just use the wrong way to log feedback. Roger Y. I also think it’s not necessary to log_evaluations manually in the evaluator. But if I just run evaluation in experiments and without log_evaluations, there is no feedback in the evaluator traces. For feedback I mean the screenshot:
James S.
·
Roger Y. I changed to evaluate the AGENT spans instead of run experiment for my requirement.
Log traces of my agent for question, answer and reference.
run_evals against the AGENT kind spans and return evaluation results.
log_evaluations for the above evaluation results.
Now there are feedbacks with explanations in each span. Now my understanding is:
Experiment evaluations are used to track the overall benchmarks.
Span evaluations are used to see the detail of feedbacks.
Correct me if I’m wrong. Thanks
Roger Y.
·
Yea i think that’s correct. Incidentally, the Feedback section in the UI also has another purpose: you can annotate manually via the “Annotate” button on the upper right hand side.
James S.
·
Roger Y. Thank you. I need feedback because I want to export the root spans with explanations to a CSV for the other team to improve the AI feature.
Roger Y.
·
I see. That’s a probably a good convenience enhancement that we can add to smooth this process for you. We appreciate you bringing this to our attention.
James S.
·
Thank you!
👍1
Roger Y.
·
can i ask what method do you use to download the root spans?

Debugging Feedback Update Issues in Arize-Phoenix 8.6.1

25 comments

Debugging Feedback Update Issues in Arize-Phoenix 8.6.1

25 comments