Troubleshooting AzureOpenAI Notebook Issues and Code Errors

ryan a. · 2024-07-09T11:51:27.915Z

Hi Jason, I hope you are doing well. I tried to run the notebook you shared with me. I am getting None for labels, score, and explanation. I am using AzureOpenAI with AzureAd cred. Please check the screenshots and the error message. Also, I do not want to use llama_index instead of AzureOpenAI client. Please take a look at the code below. I am using openai==1.29.0 from openai import AzureOpenAI openai_client = AzureOpenAI(azure_endpoint = f"https://{AZURE_OPENAI_SERVICE}.openai.azure.com (http://openai.azure.com)", api_version = OPENAI_API_VERSION, azure_ad_token_provider = token_provider, ) response = openai_client.chat.completions.create( model=CHAT_MODEL_DEPLOYMENT_NAME, # model = "deployment_name". response_format={ "type": "json_object" }, messages=[ {"role": "system", "content": """ ### Insurance Policy Information Extraction System"""} ] )

10 comments

· Sorted by Oldest

John G.
·
ryan a. - here's a tweaked version of the code I was able to get running: https://colab.research.google.com/drive/1GuohVFucnjBxFnm0yJbhYBh09W5xzpdb This is still using llamaindex's Azure clients, but should solve the empty evaluations issue you're seeing here. To switch off of llamaindex's clients, you'd need to recreate the embedding and retrieval logic they handle. Here's one example of what that looks like in Azure: https://github.com/openai/openai-cookbook/blob/main/examples/azure/chat_with_your_own_data.ipynb From the Arize/Phoenix side, you'd just want to be sure you're switching the LlamaIndexInstrumentor to use the OpenAIInstrumentor we have Let me know if that doesn't solve your issue, or you have other questions. Happy to hop on and talk through it live if that's easier!
ryan a.
·
Thanks, John G.. I really appreciate your quick response. It is working as expected using OpenAIInstrumentor. I am facing another issue. I have Questions and their Answers in the Mongo database, and responses saved in the database from a different application earlier. Now I want to create a trace using these responses. How can I log the data in the trace and evaluation of Arize from Mongo? Note: Questions and their Answers are present in the Mongo database. I found that I'll have to tracedataset for tracing, I was trying to do it with test data. I am using test dataset: traces_url = "https://storage.googleapis.com/arize-assets/phoenix/datasets/unstructured/llm/context-retrieval/trace.jsonl" that creates a trace as expected but using the code below, from phoenix.session.evaluation import get_qa_with_reference, get_retrieved_documents retrieved_documents_df = get_retrieved_documents(px.active_session()) queries_df = get_qa_with_reference(px.active_session()) it says, and there is no column "reference" and "document_score" in the retrieved_documents_df as expected, and there is no data for queries_df No retrieval documents found.
John G.
·
ryan a. - do you already have the traces present in arize, imported from mongodb? If you're just looking to load the traces and evaluations into Arize, you can either launch the app with those traces (https://docs.arize.com/phoenix/tracing/how-to-tracing/save-and-load-traces) or use the log_traces method instead (https://github.com/Arize-ai/phoenix/blob/main/src/phoenix/session/client.py#L261). The log_traces method takes in a TraceDataset object which you can create from a dataframe and include the evaluation info along with The get_retrieved_documents and get_qa_with_reference will try to pull trace data out of the active session in arize
ryan a.
·
John G. - I can run tracing using test data: https://storage.googleapis.com/arize-assets/phoenix/datasets/unstructured/llm/context-retrieval/trace.jsonl I want to use this data for evaluation but I am getting "No retrieval documents found." and an empty dataframe for "queries_df" while running the retrieved_documents_df = get_retrieved_documents(px.Client()) queries_df = get_qa_with_reference(px.Client()) Please take a look at the screenshots below. Please take a look at my arize_eval.ipynb Jupyter Notebook below.
John G.
·
Hi ryan a. - apologies for the delay there. I'm not sure why the get_qa_with_reference method isn't returning any data. That said, both these methods are just pulling and filtering spans as dataframes, so I went ahead and did the manipulation manually to unblock you while we figure this out. Check out this notebook, and let me know if you're running into any issues there:
ryan a.
·
Hi John G. - Thank you so much for the solution, I appreciate your quick response. I think your solution worked for me. Apologies for the late reply I was busy integrating the Arize App. Now, I have another requirement. This is the most required functionality we want in our Arize App. I have multiple users and want to add a filter (dropdown) in the UI of tracing beside the date filter (dropdown) for the users, based on the selection of users I want to show the OpenAI responses in tracing. Please let me know how can I add filters (dropdown) in the UI of tracing beside the date filter (dropdown). Please see the screenshots below.
John G.
·
ryan a. glad to hear that initial solution worked! In terms of the filters dropdown, we don't have a way at the moment to save custom filters. You can use the Filter condition bar there to condition on any data within the span attributes (e.g. model_name=='GPT-3.5-turbo') , but there's no way to save those filters at the moment. When you say you're trying to filter based on users, is that the user accessing the phoenix dashboard or some user parameter you're including in your traces?
ryan a.
·
John G. - I think Filter condition can work for me, let me check. Question: When you say you're trying to filter based on users, is that the user accessing the Phoenix dashboard or some user parameter you're including in your traces? Answer: Yes, I want to include a user parameter to trace. I have a separate OpenAI application that answers the questions. Then these responses will saved in the DB with the username of whoever asked the question. Now I want to show these saved responses in Arize trace and the username in a dropdown filter. Now anyone can select the username from the trace user filter then questions and answers will be visualized on the Arize UI of that user only.
John G.
·
Great - in that case, the filter conditions should work. You may have to provide the user id as metadata to make sure it's getting to Phoenix correctly if it's not already. If you're already seeing the ids in the attributes section of spans in Phoenix, you can skip that step!
ryan a.
·
Thanks John G..

John G.
·
ryan a. - here's a tweaked version of the code I was able to get running: https://colab.research.google.com/drive/1GuohVFucnjBxFnm0yJbhYBh09W5xzpdb This is still using llamaindex's Azure clients, but should solve the empty evaluations issue you're seeing here. To switch off of llamaindex's clients, you'd need to recreate the embedding and retrieval logic they handle. Here's one example of what that looks like in Azure: https://github.com/openai/openai-cookbook/blob/main/examples/azure/chat_with_your_own_data.ipynb From the Arize/Phoenix side, you'd just want to be sure you're switching the LlamaIndexInstrumentor to use the OpenAIInstrumentor we have Let me know if that doesn't solve your issue, or you have other questions. Happy to hop on and talk through it live if that's easier!
ryan a.
·
Thanks, John G.. I really appreciate your quick response. It is working as expected using OpenAIInstrumentor. I am facing another issue. I have Questions and their Answers in the Mongo database, and responses saved in the database from a different application earlier. Now I want to create a trace using these responses. How can I log the data in the trace and evaluation of Arize from Mongo? Note: Questions and their Answers are present in the Mongo database. I found that I'll have to tracedataset for tracing, I was trying to do it with test data. I am using test dataset: traces_url = "https://storage.googleapis.com/arize-assets/phoenix/datasets/unstructured/llm/context-retrieval/trace.jsonl" that creates a trace as expected but using the code below, from phoenix.session.evaluation import get_qa_with_reference, get_retrieved_documents retrieved_documents_df = get_retrieved_documents(px.active_session()) queries_df = get_qa_with_reference(px.active_session()) it says, and there is no column "reference" and "document_score" in the retrieved_documents_df as expected, and there is no data for queries_df No retrieval documents found.
John G.
·
ryan a. - do you already have the traces present in arize, imported from mongodb? If you're just looking to load the traces and evaluations into Arize, you can either launch the app with those traces (https://docs.arize.com/phoenix/tracing/how-to-tracing/save-and-load-traces) or use the log_traces method instead (https://github.com/Arize-ai/phoenix/blob/main/src/phoenix/session/client.py#L261). The log_traces method takes in a TraceDataset object which you can create from a dataframe and include the evaluation info along with The get_retrieved_documents and get_qa_with_reference will try to pull trace data out of the active session in arize
ryan a.
·
John G. - I can run tracing using test data: https://storage.googleapis.com/arize-assets/phoenix/datasets/unstructured/llm/context-retrieval/trace.jsonl I want to use this data for evaluation but I am getting "No retrieval documents found." and an empty dataframe for "queries_df" while running the retrieved_documents_df = get_retrieved_documents(px.Client()) queries_df = get_qa_with_reference(px.Client()) Please take a look at the screenshots below. Please take a look at my arize_eval.ipynb Jupyter Notebook below.
John G.
·
Hi ryan a. - apologies for the delay there. I'm not sure why the get_qa_with_reference method isn't returning any data. That said, both these methods are just pulling and filtering spans as dataframes, so I went ahead and did the manipulation manually to unblock you while we figure this out. Check out this notebook, and let me know if you're running into any issues there:
ryan a.
·
Hi John G. - Thank you so much for the solution, I appreciate your quick response. I think your solution worked for me. Apologies for the late reply I was busy integrating the Arize App. Now, I have another requirement. This is the most required functionality we want in our Arize App. I have multiple users and want to add a filter (dropdown) in the UI of tracing beside the date filter (dropdown) for the users, based on the selection of users I want to show the OpenAI responses in tracing. Please let me know how can I add filters (dropdown) in the UI of tracing beside the date filter (dropdown). Please see the screenshots below.
John G.
·
ryan a. glad to hear that initial solution worked! In terms of the filters dropdown, we don't have a way at the moment to save custom filters. You can use the Filter condition bar there to condition on any data within the span attributes (e.g. model_name=='GPT-3.5-turbo') , but there's no way to save those filters at the moment. When you say you're trying to filter based on users, is that the user accessing the phoenix dashboard or some user parameter you're including in your traces?
ryan a.
·
John G. - I think Filter condition can work for me, let me check. Question: When you say you're trying to filter based on users, is that the user accessing the Phoenix dashboard or some user parameter you're including in your traces? Answer: Yes, I want to include a user parameter to trace. I have a separate OpenAI application that answers the questions. Then these responses will saved in the DB with the username of whoever asked the question. Now I want to show these saved responses in Arize trace and the username in a dropdown filter. Now anyone can select the username from the trace user filter then questions and answers will be visualized on the Arize UI of that user only.
John G.
·
Great - in that case, the filter conditions should work. You may have to provide the user id as metadata to make sure it's getting to Phoenix correctly if it's not already. If you're already seeing the ids in the attributes section of spans in Phoenix, you can skip that step!
ryan a.
·
Thanks John G..