Ensuring Correct Trace Span Ordering in Your Project Setup

Victor L. · 2025-06-04T17:11:30.715Z

Is there a way to ensure the trace span ordering is correct? For example I have this trace and for the tool calls under AgentSupportAgent.run the completions aren't in the correct order/association with the correct tool call. Pretty basic setup with tracer_provider = register(project_name="chat", auto_instrument=True, batch=True) and using LiteLLMInstrumentor

11 comments

· Sorted by Oldest

RunLLM
APP
·
Phoenix relies on the parent-child relationships and timestamps in your spans to determine ordering, but if spans are created or ended out of order, or if context propagation is broken (especially with async code or multiple instrumentors), the UI may show tool calls and completions out of order or not properly associated. This is a known challenge, especially when mixing auto-instrumentation and manual spans, or when using frameworks that manage their own async contexts—Phoenix can only order what is correctly reported by the tracer and instrumentors according to OpenTelemetry conventions. There is no explicit Phoenix-side setting to "fix" ordering; the solution is to ensure your instrumentation and context propagation are correct and consistent across all tool calls and completions, and to avoid mixing multiple instrumentors that may not compose well together (<https://github.com/Arize-ai/phoenix/issues/7547|source>, <https://github.com/Arize-ai/phoenix/issues/6336|source>). If you are using only `LiteLLMInstrumentor` and still see ordering issues, check that all tool calls and completions are awaited in the correct order, and that you are not mixing in other instrumentors (like OpenAI or LangChain) which can cause duplicate or mis-nested spans. If spans are missing or out of order due to async context loss, you may need to manually propagate context or use synchronous code for critical sections (<https://github.com/Arize-ai/phoenix/issues/6336|source>, <https://github.com/Arize-ai/phoenix/issues/7547|source>). Would you like more detail or code examples for context propagation or debugging span order? Sources: - <https://github.com/Arize-ai/phoenix/issues/7547|https://github.com/Arize-ai/phoenix/issues/7547> - <https://github.com/Arize-ai/phoenix/issues/6336|https://github.com/Arize-ai/phoenix/issues/6336>
💡 Hint: Mention RunLLM in the thread for followups.
Xander S.
·
Hey Victor L., what tells you that the sub-spans are incorrectly ordered?
Victor L.
·
For example in my image I would expect the completion to be more easily shown/associated with its tool call. Here it's all just kind of thrown together.. I know the GetAgentPoersonalizationTool and UpdateAgentPersonalizationTool are sync so when I see this I expect the completions to line up as well
Xander S.
·
It sounds like there is a one-to-one correspondence between LLM and tool spans, and you expect them to alternate.

Victor L.

Not exactly sure what you mean by that. I just expect the tool call completion be linked in the view with it's associated tool call. For this run the AgentSupportAgent is a code agent and it was doing this

```python
# Get current personalization settings
current_settings = get_agent_personalization()

# Update the settings to unmute SMS notifications
updated_settings = update_agent_personalization(
    assistant_name=None,
    agent_name=None,
    sync_time=None,
    auxiliary_sync_time=None,
    default_appointment_duration_minutes=None,
    use_times_for_tasks=True,
    timezone=None
)

# Print the updated settings to verify
print("Updated settings:", updated_settings)

So you see both tool calls come in the but I'm expecting the completion for get_agent_personalization to be displayed before UpdateAgentPersonalizationTool

Victor L.
·
Perhaps I'm just misunderstanding the completion associations with tool calls
Mikyo
·
In general the spans are ordered by start time. It might be the case that these spans are within such a small margin of time that we can't sort at that level? Can you check the start times of the spans? cc John G. as well here.
Victor L.
·
I see. I think I'm also incorrectly associating the completions with the tools when in reality these are each thinking step for the root agent. I'm using smolagents so probably stemming from a misunderstanding of 2 frameworks simultaneously 🙂
Victor L.
·
So think we're good. Thanks for the insights
Mikyo
·
Oh got it. Don't hesitate to reach back out if you see anything confusing with smolagents. cc Dylan C.
1
Victor L.
·
Thanks. Going off the docs here https://arize.com/docs/phoenix/tracing/integrations-tracing/hfsmolagents I was expecting more of that view. I shimmed in LiteLLM to try and see if that did the trick but couldn't get the same view Look like there is an ongoing bug so my view isn't really reflecting what the demo trace has. Looks like there is a fix out https://github.com/Arize-ai/openinference/pull/1744

RunLLM
APP
·
Phoenix relies on the parent-child relationships and timestamps in your spans to determine ordering, but if spans are created or ended out of order, or if context propagation is broken (especially with async code or multiple instrumentors), the UI may show tool calls and completions out of order or not properly associated. This is a known challenge, especially when mixing auto-instrumentation and manual spans, or when using frameworks that manage their own async contexts—Phoenix can only order what is correctly reported by the tracer and instrumentors according to OpenTelemetry conventions. There is no explicit Phoenix-side setting to "fix" ordering; the solution is to ensure your instrumentation and context propagation are correct and consistent across all tool calls and completions, and to avoid mixing multiple instrumentors that may not compose well together (<https://github.com/Arize-ai/phoenix/issues/7547|source>, <https://github.com/Arize-ai/phoenix/issues/6336|source>). If you are using only `LiteLLMInstrumentor` and still see ordering issues, check that all tool calls and completions are awaited in the correct order, and that you are not mixing in other instrumentors (like OpenAI or LangChain) which can cause duplicate or mis-nested spans. If spans are missing or out of order due to async context loss, you may need to manually propagate context or use synchronous code for critical sections (<https://github.com/Arize-ai/phoenix/issues/6336|source>, <https://github.com/Arize-ai/phoenix/issues/7547|source>). Would you like more detail or code examples for context propagation or debugging span order? Sources: - <https://github.com/Arize-ai/phoenix/issues/7547|https://github.com/Arize-ai/phoenix/issues/7547> - <https://github.com/Arize-ai/phoenix/issues/6336|https://github.com/Arize-ai/phoenix/issues/6336>
💡 Hint: Mention RunLLM in the thread for followups.
Xander S.
·
Hey Victor L., what tells you that the sub-spans are incorrectly ordered?
Victor L.
·
For example in my image I would expect the completion to be more easily shown/associated with its tool call. Here it's all just kind of thrown together.. I know the GetAgentPoersonalizationTool and UpdateAgentPersonalizationTool are sync so when I see this I expect the completions to line up as well
Xander S.
·
It sounds like there is a one-to-one correspondence between LLM and tool spans, and you expect them to alternate.

Victor L.

```python
# Get current personalization settings
current_settings = get_agent_personalization()

# Update the settings to unmute SMS notifications
updated_settings = update_agent_personalization(
    assistant_name=None,
    agent_name=None,
    sync_time=None,
    auxiliary_sync_time=None,
    default_appointment_duration_minutes=None,
    use_times_for_tasks=True,
    timezone=None
)

# Print the updated settings to verify
print("Updated settings:", updated_settings)

So you see both tool calls come in the but I'm expecting the completion for get_agent_personalization to be displayed before UpdateAgentPersonalizationTool

Victor L.
·
Perhaps I'm just misunderstanding the completion associations with tool calls
Mikyo
·
In general the spans are ordered by start time. It might be the case that these spans are within such a small margin of time that we can't sort at that level? Can you check the start times of the spans? cc John G. as well here.
Victor L.
·
I see. I think I'm also incorrectly associating the completions with the tools when in reality these are each thinking step for the root agent. I'm using smolagents so probably stemming from a misunderstanding of 2 frameworks simultaneously 🙂
Victor L.
·
So think we're good. Thanks for the insights
Mikyo
·
Oh got it. Don't hesitate to reach back out if you see anything confusing with smolagents. cc Dylan C.
1
Victor L.
·
Thanks. Going off the docs here https://arize.com/docs/phoenix/tracing/integrations-tracing/hfsmolagents I was expecting more of that view. I shimmed in LiteLLM to try and see if that did the trick but couldn't get the same view Look like there is an ongoing bug so my view isn't really reflecting what the demo trace has. Looks like there is a fix out https://github.com/Arize-ai/openinference/pull/1744