Orchestrating OpenTelemetry for Service Tracing

Ben M.

·Oct 03, 2025 07:25 PM·

RunLLM How can I orchestrate opentelemetry / openinference propagation? I want to trace across services

14 comments

· Sorted by Oldest

RunLLM
APP
·
·
To trace across services with OpenTelemetry/OpenInference, you must propagate the trace context (trace ID and span ID) between services, typically via HTTP headers. The calling service injects the current trace context into outgoing requests, and the receiving service extracts this context to continue the trace, ensuring all spans are grouped under a single trace in Phoenix. This is achieved using OpenTelemetry's inject and extract methods for context propagation in your client and server code, respectively. For Python, see the following example: ``` # Client: inject context into headers from opentelemetry.propagate import inject headers = {} inject(headers) requests.post("http://service/api", headers=headers, json=payload) # Server: extract context from headers from opentelemetry.propagate import extract context = extract(request.headers) with tracer.start_as_current_span("service_span", context=context): # service logic pass ``` This approach works for any number of services and is the standard for distributed tracing with Phoenix and OpenInference. For more, see the <https://arize.com/docs/phoenix/tracing/how-to-tracing/setup-tracing/custom-spans#radix-«R25mjtt9lffffffb»|manual instrumentation guide> and <https://arize-ai.slack.com/archives/C04R3GXC8HK/p1724796536.922089|Slack discussion>. Would you like more detail or a language-specific example? Sources: - <https://arize.com/docs/phoenix/tracing/how-to-tracing/setup-tracing/custom-spans#radix-«R25mjtt9lffffffb»|https://arize.com/docs/phoenix/tracing/how-to-tracing/setup-tracing/custom-spans#radix-«R25mjtt9lffffffb»> - <https://arize-ai.slack.com/archives/C04R3GXC8HK/p1724796536.922089|https://arize-ai.slack.com/archives/C04R3GXC8HK/p1724796536.922089>
💡 Hint: Mention RunLLM in the thread for followups.
Ben M.
·
·
RunLLM does this work if the default project for each service is different? IE service a logs to a service-a project and service b logs to service-b? Will propagation work here? Does it duplicate spans to both projects?
RunLLM
APP
·
·
No, propagation will not group spans into a single trace if each service logs to a different project; each project is a separate container for traces, so the trace will be split across projects. Spans are not duplicated between projects—each span appears only in the project it was logged to, and traces are not merged across projects in Phoenix. To have a unified trace across services, all services must log to the same project. For more, see <https://arize.com/docs/phoenix/tracing/concepts-tracing/what-are-traces#user-content-traces|Phoenix tracing concepts> and <https://github.com/Arize-ai/phoenix/issues/3375|GitHub Issue #3375>. Would you like more detail or guidance on project configuration? Sources: - <https://github.com/Arize-ai/phoenix/issues/3375|https://github.com/Arize-ai/phoenix/issues/3375> - <https://arize.com/docs/phoenix/tracing/concepts-tracing/what-are-traces#user-content-traces|Phoenix tracing concepts>
💡 Hint: Mention RunLLM in the thread for followups.
Xander S.
·
·
Hey Ben M., the concept you're looking for in OTel is called "baggage" https://opentelemetry.io/docs/concepts/signals/baggage/
👀1
Ben M.
·
·
What a name 😂 I’ll look into it! Do you know off hand if spans will appear in both projects?
Xander S.
·
·
In terms of different services having different projects, I am actually not certain what will happen
Ben M.
·
·
Gotcha 👍
Xander S.
·
·
Maybe you can give it a try and let us know if something weird happens
Ben M.
·
·
Yeah, I’ll setup a quick test and report back. Should be easy to validate locally
Xander S.
·
·
We probably expect them to share the same project.
Xander S.
·
·
Is my guess
Ben M.
·
·
I setup a quick test:
I call service a and start a span, hello-world-span
This service calls service b and propagates baggage
service b starts its own span in context, api2_span and returns data to service a
Service A calls another local function, do_work which is itself a chain
Result: A service_a project does not get created and instead all work falls under service_b
Ben M.
·
·
If I use dangerously_using_project from service b, I can get everything to log under the original project though 🙂
Ben M.
·
·
Kind of an interesting paradigm. I would love it if the spans all ended up under the originating project in cases of propagation, but it’s nice knowing that we can do this at all

RunLLM
APP
·
·
To trace across services with OpenTelemetry/OpenInference, you must propagate the trace context (trace ID and span ID) between services, typically via HTTP headers. The calling service injects the current trace context into outgoing requests, and the receiving service extracts this context to continue the trace, ensuring all spans are grouped under a single trace in Phoenix. This is achieved using OpenTelemetry's inject and extract methods for context propagation in your client and server code, respectively. For Python, see the following example: ``` # Client: inject context into headers from opentelemetry.propagate import inject headers = {} inject(headers) requests.post("http://service/api", headers=headers, json=payload) # Server: extract context from headers from opentelemetry.propagate import extract context = extract(request.headers) with tracer.start_as_current_span("service_span", context=context): # service logic pass ``` This approach works for any number of services and is the standard for distributed tracing with Phoenix and OpenInference. For more, see the <https://arize.com/docs/phoenix/tracing/how-to-tracing/setup-tracing/custom-spans#radix-«R25mjtt9lffffffb»|manual instrumentation guide> and <https://arize-ai.slack.com/archives/C04R3GXC8HK/p1724796536.922089|Slack discussion>. Would you like more detail or a language-specific example? Sources: - <https://arize.com/docs/phoenix/tracing/how-to-tracing/setup-tracing/custom-spans#radix-«R25mjtt9lffffffb»|https://arize.com/docs/phoenix/tracing/how-to-tracing/setup-tracing/custom-spans#radix-«R25mjtt9lffffffb»> - <https://arize-ai.slack.com/archives/C04R3GXC8HK/p1724796536.922089|https://arize-ai.slack.com/archives/C04R3GXC8HK/p1724796536.922089>
💡 Hint: Mention RunLLM in the thread for followups.
Ben M.
·
·
RunLLM does this work if the default project for each service is different? IE service a logs to a service-a project and service b logs to service-b? Will propagation work here? Does it duplicate spans to both projects?
RunLLM
APP
·
·
No, propagation will not group spans into a single trace if each service logs to a different project; each project is a separate container for traces, so the trace will be split across projects. Spans are not duplicated between projects—each span appears only in the project it was logged to, and traces are not merged across projects in Phoenix. To have a unified trace across services, all services must log to the same project. For more, see <https://arize.com/docs/phoenix/tracing/concepts-tracing/what-are-traces#user-content-traces|Phoenix tracing concepts> and <https://github.com/Arize-ai/phoenix/issues/3375|GitHub Issue #3375>. Would you like more detail or guidance on project configuration? Sources: - <https://github.com/Arize-ai/phoenix/issues/3375|https://github.com/Arize-ai/phoenix/issues/3375> - <https://arize.com/docs/phoenix/tracing/concepts-tracing/what-are-traces#user-content-traces|Phoenix tracing concepts>
💡 Hint: Mention RunLLM in the thread for followups.
Xander S.
·
·
Hey Ben M., the concept you're looking for in OTel is called "baggage" https://opentelemetry.io/docs/concepts/signals/baggage/
👀1
Ben M.
·
·
What a name 😂 I’ll look into it! Do you know off hand if spans will appear in both projects?
Xander S.
·
·
In terms of different services having different projects, I am actually not certain what will happen
Ben M.
·
·
Gotcha 👍
Xander S.
·
·
Maybe you can give it a try and let us know if something weird happens
Ben M.
·
·
Yeah, I’ll setup a quick test and report back. Should be easy to validate locally
Xander S.
·
·
We probably expect them to share the same project.
Xander S.
·
·
Is my guess
Ben M.
·
·
I setup a quick test:
I call service a and start a span, hello-world-span
This service calls service b and propagates baggage
service b starts its own span in context, api2_span and returns data to service a
Service A calls another local function, do_work which is itself a chain
Result: A service_a project does not get created and instead all work falls under service_b
Ben M.
·
·
If I use dangerously_using_project from service b, I can get everything to log under the original project though 🙂
Ben M.
·
·
Kind of an interesting paradigm. I would love it if the spans all ended up under the originating project in cases of propagation, but it’s nice knowing that we can do this at all