How does the Trace Details view decide which spans to include? Recently we've run into an issue where clicking on a root span opens Trace Details with multiple other root spans included. The other root spans are from tasks started by the same worker in our app, but the actual spans are each created in separate child worker contexts, so I'm not sure how these are being grouped like this. 馃
馃挕聽Hint: Mention RunLLM in the thread for followups.
馃挕聽Hint: Mention RunLLM in the thread for followups.
OK, points me in a general direction at least. Thanks.
I think this might be an interaction between our opentelemetry for perf analysis and openinference for llm analysis. The telemetry code for our worker library appears to be serializing context from the enqueue side and setting it on the worker task. Maybe I can reset the openinference trace id in the child tasks. 馃 In some ways it's useful to have them grouped this way, but it's making navigation in the phoenix UI more challenging right now. 馃槄
It looks like the opentelemetry code is setting a "traceparent". Anyone happen to know of a way I could erase that for openinference, without affecting our other tracing?
It looks like I might be able to erase it completely with something similar to this example, but it would be cool if I could control when the two telemetry purposes intersect. https://opentelemetry.io/docs/languages/python/cookbook/#manually-setting-span-context
This is what I ended up doing for now:
token = context.attach(trace.set_span_in_context(INVALID_SPAN))
try:
// llm stuff
finally:
context.detach(token)In this case, the hierarchy isn't coming from llama-index, it's coming from the opentelemetry package we use for arq (a python worker library). We have that package installed for perf tracing, and the perf spans caused the llm spans to be grouped in an unintended way.
