Guidance Needed: Instrumenting Retrieval in Azure CosmosDB MongoDB vCore
Hi team! Long time lurker and big fan of your work. Newbie RAG instrumentation question: how would I go about instrumenting retrieval from an Azure CosmosDB Mongo vCore vector index? I have:
A working deployment of Microsoft's Sample Chat App with AOAI, forked from https://github.com/microsoft/sample-app-aoai-chatGPT. (this is the repo that powers the "deploy... as a web app" button available when you view a model deployment in the Azure OpenAI Studio)
An Azure Cosmos DB MongoDB vCore vector index, loaded with document chunks from a single PDF, and from which the web app successfully retrieves chunks
A hosted Phoenix instance at https://app.phoenix.arize.com/ for testing purposes
I followed the Python instrumentation instructions shown by clicking "Get Started" in a new Phoenix project, by pip installing openinference-instrumentation-openai and adding this near the top of app.py:
from phoenix.otel import register
from openinference.instrumentation.openai import OpenAIInstrumentor
tracer_provider = register(
project_name=project_name, # I populated my actual project name here
endpoint="https://app.phoenix.arize.com/v1/traces"
)
OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)
These changes allow traces to be sent into Phoenix, but they're all ChatCompletions, including those with document retrieval. I can see the retrieved document chunks in the trace's Output, and model invocation parameters under the Attributes tab. Do you have guidance on how to better instrument the retrieval step of this web app?
