Sanjana Y.

Hi chezky K. From my understanding, seems like you have a ground-truth dataset and you want to use it to benchmark how well your RAG system performs. You can use either Phoenix or AX for this. The workflow would involve uploading that dataset to the platform and setting up an experiment that includes an eval to see how output from your system matches the expected output (from the ground truth dataset). This Phoenux cookbook doesn't match your exact use case BUT it does address using an eval to see how an output matches your ground truth: https://arize.com/docs/phoenix/cookbook/ai-engineering-workflows/creating-a-custom-llm-evaluator-with-a-benchmark-dataset

Hey Jayakrishna Y., the issue seems to be that in the AssistantAgent you are setting model_client_stream=True . If you remove this, you will be able to see tokens & cost for each LLM span in Phoenix. One more thing, I noticed in your dependencies list, you don't have openinference-instrumentation-autogen-agentchat . Make sure this is installed so you can see the full trace. You can also remove the 2 lines below because setting auto_instrument=True takes care of it for you: from openinference.instrumentation.openai import OpenAIInstrumentor OpenAIInstrumentor().instrument(tracer_provider=tracer_provider) Let me know if you need anymore help with this!

Hey Jayakrishna Y., was able to reproduce your error. Thanks for sharing the code snippets. Working on the solution - will keep you updated

taking a look again

the endpoint shouldnt be the issue here

Did you navigate to settings and make a new API key

https://github.com/Arize-ai/openinference/issues/1891

Brandon Not at the moment, but is there a specific workflow you’re looking to enable with API access?