Asynchronous LLM Calls: Trace Differences in Phoenix Support

Dvir B. · 2025-07-29T08:51:01.295Z

Hi #phoenix-support , we're using phoenix for a while now. recently, we built a new app backed by llama-index. Some of the LLM calls there are wrapped by either FunctionCallingProgram or OpenAIPydanticProgram, for working with structured responses. We've noticed that when these programs are called regularly, we can view the named parameters (or inputs) to the program through the traces. However, when we call these program asynchronously, using their acall method, these inputs are not exposed as part of the trace, and the only thing we have there is the formatted template prompt already including the relevant parameters (without being able to see which parameters were sent). Is that a behaviour your familiar with? that asynchronous calls are traced differently from "regular" calls? the implementations of the calls on both cases is identical except for the async llm call.

Asynchronous LLM Calls: Trace Differences in Phoenix Support | Arize AI Community

6 comments

· Sorted by Oldest

RunLLM
APP
·
Yes, this is a known limitation: in Phoenix, some async LlamaIndex workflows (including those using acall on FunctionCallingProgram or OpenAIPydanticProgram) do not currently trace or expose invocation parameters as cleanly as their synchronous counterparts. This gap in async tracing has been observed and reported, and is due to incomplete async instrumentation in the current OpenInference/LlamaIndex integration—async calls may result in missing or incomplete span attributes, including input parameters, in the Phoenix UI. Synchronous calls, by contrast, typically show all invocation parameters as expected in the trace details [fa606d08].If you need full traceability of input parameters for async calls, this is not yet fully supported and may require updates to the OpenInference instrumentation or LlamaIndex itself. The Phoenix and OpenInference teams are aware of this gap and are working on improving async tracing coverage [fa606d08].Would you like more detail or guidance on possible workarounds?Sources: - fa606d08
💡 Hint: Mention RunLLM in the thread for followups.
Dvir B.
·
RunLLM yes
RunLLM
APP
·
Step-by-step reasoning: 1. Synchronous vs. Asynchronous Tracing: In LlamaIndex, when you use synchronous program calls (e.g., call), the OpenInference/Phoenix instrumentation hooks into the call stack and captures both the formatted prompt and the original input parameters, which are then attached as attributes to the trace span. This allows Phoenix to display both the prompt and the named parameters in the UI [672765d3].2. Async Instrumentation Gaps: For asynchronous calls (e.g., acall), the tracing relies on async-compatible hooks. However, as of recent versions, the OpenInference and LlamaIndex integration does not always propagate or capture the input parameters in the same way for async calls. This is a known gap and has been discussed in the context of both missing spans and missing attributes for async workflows [672765d3; 6b83b9f5].3. Root Cause: The root cause is that the callback or instrumentor responsible for capturing the parameters is not always invoked or does not have access to the original arguments in the async path. This can be due to context propagation issues, missing callback manager wiring, or incomplete async support in the tracing handler [672765d3].4. Workarounds: - Manual Tracing: As a workaround, you can manually create a span around your async call and attach the parameters as attributes using OpenTelemetry or Phoenix's manual tracing helpers. This ensures the parameters are included in the trace, even if the automatic instrumentation misses them [672765d3]. - Callback Manager: Ensure that the callback manager (with the OpenInferenceTraceCallbackHandler) is explicitly passed to all components, including the async program and any subcomponents, as sometimes the global handler is not propagated in async contexts [672765d3]. - Version Updates: Monitor updates to the OpenInference and LlamaIndex packages, as recent releases have improved async tracing support. Upgrading may resolve some issues, but full parity is not guaranteed yet [6b83b9f5].5. Reporting and Tracking: If you need this feature, consider filing or upvoting issues in the OpenInference and LlamaIndex repositories to help prioritize async tracing improvements.Let me know if you want a code example for manual async tracing.Sources: - https://github.com/Arize-ai/phoenix/issues/1700 - https://github.com/Arize-ai/phoenix/issues/3274
💡 Hint: Mention RunLLM in the thread for followups.
RunLLM
APP
·
If you tell me the right answer to this question, I'll make sure I don't make the same mistake twice!
Feedback
Type the right answer here...
↵ Please 'enter' to submit
Mikyo
·
Dvir B. - Good morning. I'm not sure the AI bot is correct here but I can provide some context into how llma-index instrumentation works. It actually converts llma-index instrumentation spans - so the data in the spans comes directly from the llama-index package itself (e.g. what they deem important). So if data is lacking it could be due to the way in which the llama-index package is setup. Because of this, it can be worth engaging the llama-index team as they might be the most knowledgeable about their own framework. Happy to help coordinate the fix on the openinference side https://github.com/Arize-ai/openinference Thanks for the feedback! I can also help with filing tickets with llama-index and escalating with Logan and others if you get stuck.
👍
Dvir B.
·
Thanks a lot Mikyo, we'll take a look into int

RunLLM
APP
·
Yes, this is a known limitation: in Phoenix, some async LlamaIndex workflows (including those using acall on FunctionCallingProgram or OpenAIPydanticProgram) do not currently trace or expose invocation parameters as cleanly as their synchronous counterparts. This gap in async tracing has been observed and reported, and is due to incomplete async instrumentation in the current OpenInference/LlamaIndex integration—async calls may result in missing or incomplete span attributes, including input parameters, in the Phoenix UI. Synchronous calls, by contrast, typically show all invocation parameters as expected in the trace details [fa606d08].If you need full traceability of input parameters for async calls, this is not yet fully supported and may require updates to the OpenInference instrumentation or LlamaIndex itself. The Phoenix and OpenInference teams are aware of this gap and are working on improving async tracing coverage [fa606d08].Would you like more detail or guidance on possible workarounds?Sources: - fa606d08
💡 Hint: Mention RunLLM in the thread for followups.
Dvir B.
·
RunLLM yes
RunLLM
APP
·
Step-by-step reasoning: 1. Synchronous vs. Asynchronous Tracing: In LlamaIndex, when you use synchronous program calls (e.g., call), the OpenInference/Phoenix instrumentation hooks into the call stack and captures both the formatted prompt and the original input parameters, which are then attached as attributes to the trace span. This allows Phoenix to display both the prompt and the named parameters in the UI [672765d3].2. Async Instrumentation Gaps: For asynchronous calls (e.g., acall), the tracing relies on async-compatible hooks. However, as of recent versions, the OpenInference and LlamaIndex integration does not always propagate or capture the input parameters in the same way for async calls. This is a known gap and has been discussed in the context of both missing spans and missing attributes for async workflows [672765d3; 6b83b9f5].3. Root Cause: The root cause is that the callback or instrumentor responsible for capturing the parameters is not always invoked or does not have access to the original arguments in the async path. This can be due to context propagation issues, missing callback manager wiring, or incomplete async support in the tracing handler [672765d3].4. Workarounds: - Manual Tracing: As a workaround, you can manually create a span around your async call and attach the parameters as attributes using OpenTelemetry or Phoenix's manual tracing helpers. This ensures the parameters are included in the trace, even if the automatic instrumentation misses them [672765d3]. - Callback Manager: Ensure that the callback manager (with the OpenInferenceTraceCallbackHandler) is explicitly passed to all components, including the async program and any subcomponents, as sometimes the global handler is not propagated in async contexts [672765d3]. - Version Updates: Monitor updates to the OpenInference and LlamaIndex packages, as recent releases have improved async tracing support. Upgrading may resolve some issues, but full parity is not guaranteed yet [6b83b9f5].5. Reporting and Tracking: If you need this feature, consider filing or upvoting issues in the OpenInference and LlamaIndex repositories to help prioritize async tracing improvements.Let me know if you want a code example for manual async tracing.Sources: - https://github.com/Arize-ai/phoenix/issues/1700 - https://github.com/Arize-ai/phoenix/issues/3274
💡 Hint: Mention RunLLM in the thread for followups.
RunLLM
APP
·
If you tell me the right answer to this question, I'll make sure I don't make the same mistake twice!
Feedback
Type the right answer here...
↵ Please 'enter' to submit
Mikyo
·
Dvir B. - Good morning. I'm not sure the AI bot is correct here but I can provide some context into how llma-index instrumentation works. It actually converts llma-index instrumentation spans - so the data in the spans comes directly from the llama-index package itself (e.g. what they deem important). So if data is lacking it could be due to the way in which the llama-index package is setup. Because of this, it can be worth engaging the llama-index team as they might be the most knowledgeable about their own framework. Happy to help coordinate the fix on the openinference side https://github.com/Arize-ai/openinference Thanks for the feedback! I can also help with filing tickets with llama-index and escalating with Logan and others if you get stuck.
👍
Dvir B.
·
Thanks a lot Mikyo, we'll take a look into int