Workaround Needed for AzureChatOpenAI Concurrency Issue in Evaluations

Akshay P. · 2025-04-08T04:56:25.497Z

Hi Folks for Hallucination and Ref evaluator should we use eval_model = OpenAIModel( model="gpt-4-turbo-preview", ) Because I have AzureChatOpenAI key and If I use it for evals its throwing 'AzureChatOpenAI' object has no attribute 'default_concurrency' I tried patching it eval_model.default_concurrency = 1 it didnt work any workaround for this?

Workaround Needed for AzureChatOpenAI Concurrency Issue in Evaluations | Arize AI Community

13 comments

· Sorted by Oldest

RunLLM
APP
·
It seems likely that the issue arises because the `AzureChatOpenAI` object is not directly supported by Phoenix's evaluation framework. Instead, you should use the `OpenAIModel` class with Azure-specific parameters. According to <https://docs.arize.com/phoenix/api/evaluation-models#%3AR1cd9uucqfkvfa%3A|Phoenix documentation>, you can configure the `OpenAIModel` for Azure by specifying the `azure_endpoint` and `api_version` parameters. Here's an example of how you might set it up: ``` from phoenix.evals import OpenAIModel eval_model = OpenAIModel( model="gpt-4-turbo-preview", azure_endpoint="https://your-endpoint.azure.com/", api_version="2023-09-15-preview", api_key="your-azure-api-key" ) ``` This setup should allow you to use the `OpenAIModel` with Azure OpenAI for evaluations without encountering the `default_concurrency` attribute error. Would you like more details on setting up the model or running evaluations? Sources: - <https://docs.arize.com/phoenix/api/evaluation-models#%3AR1cd9uucqfkvfa%3A|Phoenix documentation>
💡 Hint: Mention RunLLM in the thread for followups.
Akshay P.
·
RunLLM Exception in worker on attempt 1: raised NotFoundError("Error code: 404 - {'error': {'code': 'DeploymentNotFound', 'message': 'The API deployment for this resource does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.'}}")
RunLLM
APP
·
The error message indicates that the specified API deployment for your Azure OpenAI resource could not be found. This could be due to a few reasons: 1. *Incorrect Deployment Name*: Ensure that the deployment name specified in your `OpenAIModel` configuration matches exactly with the deployment name in your Azure OpenAI resource. According to <https://github.com/Arize-ai/phoenix/blob/main/docs/api/evaluation-models.md|Phoenix documentation>, the `model` parameter should be set to the deployment name. 2. *Recent Deployment*: If you have just created the deployment, it might take a few minutes for it to become active. The error message suggests waiting a few minutes and trying again. Would you like more details on how to verify your deployment settings or troubleshoot further? Sources: - <https://github.com/Arize-ai/phoenix/blob/main/docs/api/evaluation-models.md|Phoenix documentation>
💡 Hint: Mention RunLLM in the thread for followups.
Akshay P.
·
RunLLM eval_model = OpenAIModel( model="gpt-4o", azure_endpoint="some-endpoint", api_version="2025-01-01-preview", api_key="222222" )
RunLLM
APP
·
The configuration you provided for the `OpenAIModel` seems mostly correct, but ensure that the `model` parameter is set to the correct deployment name as configured in your Azure OpenAI resource. The `azure_endpoint` should be the full URL of your Azure OpenAI endpoint, and the `api_version` should match the version supported by your deployment. Here's a checklist to ensure everything is set up correctly: 1. *Model Name*: Ensure that the `model` parameter matches the deployment name in Azure. This is crucial as it identifies the specific model deployment you want to use. 2. *Azure Endpoint*: The `azure_endpoint` should be the full URL provided by Azure for your OpenAI resource. It typically looks like `https://<your-resource-name> .openai.azure.com/`. 3. *API Version*: Verify that the `api_version` is correct and supported by your Azure deployment. You can check the available versions in the Azure portal or documentation. 4. *API Key*: Ensure that the `api_key` is correct and has the necessary permissions to access the Azure OpenAI resource. If you continue to face issues, double-check these parameters in the Azure portal to ensure they match your configuration. Would you like more detailed guidance on any of these steps? Sources: - <https://github.com/Arize-ai/phoenix/blob/main/docs/api/evaluation-models.md|Phoenix documentation>
💡 Hint: Mention RunLLM in the thread for followups.
Akshay P.
·
RunLLM thank you issue is solved
🙌1
RunLLM
APP
·
I'm glad to hear that your issue is resolved! If you have any more questions or need further assistance with Phoenix or related topics, feel free to ask.
💡 Hint: Mention RunLLM in the thread for followups.
Akshay P.
·
RunLLM my chatbot is api can I trace request and response of api which is hosted on azure using Phoenix
RunLLM
APP
·
Yes, you can trace the request and response of your API hosted on Azure using Phoenix. To do this, you need to manually instrument your application using OpenTelemetry, which Phoenix supports. Here's a general approach: 1. *Set Up OpenTelemetry*: Install the necessary OpenTelemetry packages in your application. This includes the OpenTelemetry SDK and an exporter to send traces to Phoenix. 2. *Configure Tracing*: Set up a tracer provider in your application to capture and export traces. You can use the `OTLPSpanExporter` to send traces to a Phoenix instance. 3. *Instrument Your API*: Use the OpenTelemetry API to create spans for your API requests and responses. This involves wrapping your API logic with spans to capture relevant data. 4. *Send Traces to Phoenix*: Ensure your traces are sent to a Phoenix instance running locally or in the cloud. You can configure the endpoint in the `OTLPSpanExporter`. Would you like more detailed guidance on setting up OpenTelemetry for your specific use case? Sources: - <https://github.com/Arize-ai/openinference/blob/main/python/instrumentation/openinference-instrumentation-openai/README.md|https://github.com/Arize-ai/openinference/blob/main/python/instrumentation/openinference-instrumentation-openai/README.md> - <https://github.com/Arize-ai/openinference/blob/main/python/instrumentation/openinference-instrumentation-haystack/README.md|https://github.com/Arize-ai/openinference/blob/main/python/instrumentation/openinference-instrumentation-haystack/README.md>
💡 Hint: Mention RunLLM in the thread for followups.
Akshay P.
·
RunLLM I have endpoint url as well
RunLLM
APP
·
To trace your API using Phoenix with the endpoint URL you have, follow these steps: 1. *Install OpenTelemetry and Phoenix Packages*: Ensure you have the necessary packages installed in your application environment. ``` pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http arize-phoenix ``` 2. *Configure the Tracer*: Set up the OpenTelemetry tracer to send traces to your Phoenix endpoint. ``` from opentelemetry import trace from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import SimpleSpanProcessor # Set your Phoenix endpoint endpoint = "http://your-phoenix-endpoint/v1/traces" # Configure the tracer provider tracer_provider = TracerProvider() span_exporter = OTLPSpanExporter(endpoint=endpoint) span_processor = SimpleSpanProcessor(span_exporter) tracer_provider.add_span_processor(span_processor) trace.set_tracer_provider(tracer_provider) # Get a tracer tracer = trace.get_tracer(__name__) ``` 3. *Instrument Your API*: Use the tracer to create spans around your API logic. ``` from flask import Flask, request app = Flask(__name__) @app.route('/your-api-endpoint', methods=['POST']) def your_api_function(): with tracer.start_as_current_span("api_request") as span: # Add request details to the span span.set_attribute("http.method", request.method) span.set_attribute("http.url", request.url) # Your API logic here response = "Your response" # Add response details to the span span.set_attribute("http.status_code", 200) return response if __name__ == "__main__": app.run(host='0.0.0.0', port=5000) ``` 4. *Run Your Application*: Start your application and ensure it sends traces to the Phoenix endpoint. This setup will allow you to trace requests and responses of your API using Phoenix. Would you like more details on any specific part of this setup? Sources: - <https://github.com/Arize-ai/openinference/blob/main/python/instrumentation/openinference-instrumentation-haystack/README.md|Phoenix Documentation>
💡 Hint: Mention RunLLM in the thread for followups.
Akshay P.
·
RunLLM this is sample request results = requests.post(url="https://chatbot.com",data=json.dumps(input_data)) results
RunLLM
APP
·
To trace the request and response of your API call using Phoenix, you can manually instrument the code with OpenTelemetry. Here's how you can do it: 1. *Set Up OpenTelemetry*: Ensure you have the necessary OpenTelemetry packages installed. ``` pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http ``` 2. *Configure the Tracer*: Set up the OpenTelemetry tracer to send traces to your Phoenix endpoint. ``` from opentelemetry import trace from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import SimpleSpanProcessor # Set your Phoenix endpoint endpoint = "http://your-phoenix-endpoint/v1/traces" # Configure the tracer provider tracer_provider = TracerProvider() span_exporter = OTLPSpanExporter(endpoint=endpoint) span_processor = SimpleSpanProcessor(span_exporter) tracer_provider.add_span_processor(span_processor) trace.set_tracer_provider(tracer_provider) # Get a tracer tracer = trace.get_tracer(__name__) ``` 3. *Instrument the API Call*: Use the tracer to create spans around your API call. ``` import requests import json url = "https://chatbot.com" input_data = {"key": "value"} with tracer.start_as_current_span("api_request") as span: # Add request details to the span span.set_attribute("http.method", "POST") span.set_attribute("http.url", url) # Make the API call response = requests.post(url, data=json.dumps(input_data)) # Add response details to the span span.set_attribute("http.status_code", response.status_code) span.set_attribute("response.body", response.text) print(response.text) ``` This setup will allow you to trace the request and response of your API call using Phoenix. Would you like more details on any specific part of this setup? Sources: - <https://github.com/Arize-ai/openinference/blob/main/python/examples/dspy-rag-fastapi/frontend/pages/3_optimize_pipeline.py|https://github.com/Arize-ai/openinference/blob/main/python/examples/dspy-rag-fastapi/frontend/pages/3_optimize_pipeline.py> - <https://github.com/Arize-ai/phoenix/blob/main/examples/manually-instrumented-chatbot/frontend/src/app.py|https://github.com/Arize-ai/phoenix/blob/main/examples/manually-instrumented-chatbot/frontend/src/app.py>
💡 Hint: Mention RunLLM in the thread for followups.

RunLLM
APP
·
It seems likely that the issue arises because the `AzureChatOpenAI` object is not directly supported by Phoenix's evaluation framework. Instead, you should use the `OpenAIModel` class with Azure-specific parameters. According to <https://docs.arize.com/phoenix/api/evaluation-models#%3AR1cd9uucqfkvfa%3A|Phoenix documentation>, you can configure the `OpenAIModel` for Azure by specifying the `azure_endpoint` and `api_version` parameters. Here's an example of how you might set it up: ``` from phoenix.evals import OpenAIModel eval_model = OpenAIModel( model="gpt-4-turbo-preview", azure_endpoint="https://your-endpoint.azure.com/", api_version="2023-09-15-preview", api_key="your-azure-api-key" ) ``` This setup should allow you to use the `OpenAIModel` with Azure OpenAI for evaluations without encountering the `default_concurrency` attribute error. Would you like more details on setting up the model or running evaluations? Sources: - <https://docs.arize.com/phoenix/api/evaluation-models#%3AR1cd9uucqfkvfa%3A|Phoenix documentation>
💡 Hint: Mention RunLLM in the thread for followups.
Akshay P.
·
RunLLM Exception in worker on attempt 1: raised NotFoundError("Error code: 404 - {'error': {'code': 'DeploymentNotFound', 'message': 'The API deployment for this resource does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.'}}")
RunLLM
APP
·
The error message indicates that the specified API deployment for your Azure OpenAI resource could not be found. This could be due to a few reasons: 1. *Incorrect Deployment Name*: Ensure that the deployment name specified in your `OpenAIModel` configuration matches exactly with the deployment name in your Azure OpenAI resource. According to <https://github.com/Arize-ai/phoenix/blob/main/docs/api/evaluation-models.md|Phoenix documentation>, the `model` parameter should be set to the deployment name. 2. *Recent Deployment*: If you have just created the deployment, it might take a few minutes for it to become active. The error message suggests waiting a few minutes and trying again. Would you like more details on how to verify your deployment settings or troubleshoot further? Sources: - <https://github.com/Arize-ai/phoenix/blob/main/docs/api/evaluation-models.md|Phoenix documentation>
💡 Hint: Mention RunLLM in the thread for followups.
Akshay P.
·
RunLLM eval_model = OpenAIModel( model="gpt-4o", azure_endpoint="some-endpoint", api_version="2025-01-01-preview", api_key="222222" )
RunLLM
APP
·
The configuration you provided for the `OpenAIModel` seems mostly correct, but ensure that the `model` parameter is set to the correct deployment name as configured in your Azure OpenAI resource. The `azure_endpoint` should be the full URL of your Azure OpenAI endpoint, and the `api_version` should match the version supported by your deployment. Here's a checklist to ensure everything is set up correctly: 1. *Model Name*: Ensure that the `model` parameter matches the deployment name in Azure. This is crucial as it identifies the specific model deployment you want to use. 2. *Azure Endpoint*: The `azure_endpoint` should be the full URL provided by Azure for your OpenAI resource. It typically looks like `https://<your-resource-name> .openai.azure.com/`. 3. *API Version*: Verify that the `api_version` is correct and supported by your Azure deployment. You can check the available versions in the Azure portal or documentation. 4. *API Key*: Ensure that the `api_key` is correct and has the necessary permissions to access the Azure OpenAI resource. If you continue to face issues, double-check these parameters in the Azure portal to ensure they match your configuration. Would you like more detailed guidance on any of these steps? Sources: - <https://github.com/Arize-ai/phoenix/blob/main/docs/api/evaluation-models.md|Phoenix documentation>
💡 Hint: Mention RunLLM in the thread for followups.
Akshay P.
·
RunLLM thank you issue is solved
🙌1
RunLLM
APP
·
I'm glad to hear that your issue is resolved! If you have any more questions or need further assistance with Phoenix or related topics, feel free to ask.
💡 Hint: Mention RunLLM in the thread for followups.
Akshay P.
·
RunLLM my chatbot is api can I trace request and response of api which is hosted on azure using Phoenix
RunLLM
APP
·
Yes, you can trace the request and response of your API hosted on Azure using Phoenix. To do this, you need to manually instrument your application using OpenTelemetry, which Phoenix supports. Here's a general approach: 1. *Set Up OpenTelemetry*: Install the necessary OpenTelemetry packages in your application. This includes the OpenTelemetry SDK and an exporter to send traces to Phoenix. 2. *Configure Tracing*: Set up a tracer provider in your application to capture and export traces. You can use the `OTLPSpanExporter` to send traces to a Phoenix instance. 3. *Instrument Your API*: Use the OpenTelemetry API to create spans for your API requests and responses. This involves wrapping your API logic with spans to capture relevant data. 4. *Send Traces to Phoenix*: Ensure your traces are sent to a Phoenix instance running locally or in the cloud. You can configure the endpoint in the `OTLPSpanExporter`. Would you like more detailed guidance on setting up OpenTelemetry for your specific use case? Sources: - <https://github.com/Arize-ai/openinference/blob/main/python/instrumentation/openinference-instrumentation-openai/README.md|https://github.com/Arize-ai/openinference/blob/main/python/instrumentation/openinference-instrumentation-openai/README.md> - <https://github.com/Arize-ai/openinference/blob/main/python/instrumentation/openinference-instrumentation-haystack/README.md|https://github.com/Arize-ai/openinference/blob/main/python/instrumentation/openinference-instrumentation-haystack/README.md>
💡 Hint: Mention RunLLM in the thread for followups.
Akshay P.
·
RunLLM I have endpoint url as well
RunLLM
APP
·
To trace your API using Phoenix with the endpoint URL you have, follow these steps: 1. *Install OpenTelemetry and Phoenix Packages*: Ensure you have the necessary packages installed in your application environment. ``` pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http arize-phoenix ``` 2. *Configure the Tracer*: Set up the OpenTelemetry tracer to send traces to your Phoenix endpoint. ``` from opentelemetry import trace from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import SimpleSpanProcessor # Set your Phoenix endpoint endpoint = "http://your-phoenix-endpoint/v1/traces" # Configure the tracer provider tracer_provider = TracerProvider() span_exporter = OTLPSpanExporter(endpoint=endpoint) span_processor = SimpleSpanProcessor(span_exporter) tracer_provider.add_span_processor(span_processor) trace.set_tracer_provider(tracer_provider) # Get a tracer tracer = trace.get_tracer(__name__) ``` 3. *Instrument Your API*: Use the tracer to create spans around your API logic. ``` from flask import Flask, request app = Flask(__name__) @app.route('/your-api-endpoint', methods=['POST']) def your_api_function(): with tracer.start_as_current_span("api_request") as span: # Add request details to the span span.set_attribute("http.method", request.method) span.set_attribute("http.url", request.url) # Your API logic here response = "Your response" # Add response details to the span span.set_attribute("http.status_code", 200) return response if __name__ == "__main__": app.run(host='0.0.0.0', port=5000) ``` 4. *Run Your Application*: Start your application and ensure it sends traces to the Phoenix endpoint. This setup will allow you to trace requests and responses of your API using Phoenix. Would you like more details on any specific part of this setup? Sources: - <https://github.com/Arize-ai/openinference/blob/main/python/instrumentation/openinference-instrumentation-haystack/README.md|Phoenix Documentation>
💡 Hint: Mention RunLLM in the thread for followups.
Akshay P.
·
RunLLM this is sample request results = requests.post(url="https://chatbot.com",data=json.dumps(input_data)) results
RunLLM
APP
·
To trace the request and response of your API call using Phoenix, you can manually instrument the code with OpenTelemetry. Here's how you can do it: 1. *Set Up OpenTelemetry*: Ensure you have the necessary OpenTelemetry packages installed. ``` pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http ``` 2. *Configure the Tracer*: Set up the OpenTelemetry tracer to send traces to your Phoenix endpoint. ``` from opentelemetry import trace from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import SimpleSpanProcessor # Set your Phoenix endpoint endpoint = "http://your-phoenix-endpoint/v1/traces" # Configure the tracer provider tracer_provider = TracerProvider() span_exporter = OTLPSpanExporter(endpoint=endpoint) span_processor = SimpleSpanProcessor(span_exporter) tracer_provider.add_span_processor(span_processor) trace.set_tracer_provider(tracer_provider) # Get a tracer tracer = trace.get_tracer(__name__) ``` 3. *Instrument the API Call*: Use the tracer to create spans around your API call. ``` import requests import json url = "https://chatbot.com" input_data = {"key": "value"} with tracer.start_as_current_span("api_request") as span: # Add request details to the span span.set_attribute("http.method", "POST") span.set_attribute("http.url", url) # Make the API call response = requests.post(url, data=json.dumps(input_data)) # Add response details to the span span.set_attribute("http.status_code", response.status_code) span.set_attribute("response.body", response.text) print(response.text) ``` This setup will allow you to trace the request and response of your API call using Phoenix. Would you like more details on any specific part of this setup? Sources: - <https://github.com/Arize-ai/openinference/blob/main/python/examples/dspy-rag-fastapi/frontend/pages/3_optimize_pipeline.py|https://github.com/Arize-ai/openinference/blob/main/python/examples/dspy-rag-fastapi/frontend/pages/3_optimize_pipeline.py> - <https://github.com/Arize-ai/phoenix/blob/main/examples/manually-instrumented-chatbot/frontend/src/app.py|https://github.com/Arize-ai/phoenix/blob/main/examples/manually-instrumented-chatbot/frontend/src/app.py>
💡 Hint: Mention RunLLM in the thread for followups.