Arize AI Community Icon

Manual Instrumentation Issues: Spans Not Showing in Phoenix Dashboard | Arize AI Community

Phoenix Support

Manual Instrumentation Issues: Spans Not Showing in Phoenix Dashboard

·Oct 10, 2024 08:03 PM

hi! i am trying to mix manual instrumentation my application with spans from llama-index, but on the phoenix dashboard none of my manual spans are showing up. i've peppered in dbg statements through open telemetry, and my manually created spans are being exported successfully (status code 204). i can see the llama-index spans but my manually createdspans do not show up at all.

# init code
def init_tracer(spawn_server: bool = True):
    resource = Resource(attributes={ResourceAttributes.PROJECT_NAME: "viv4"})
    tracer_provider = TracerProvider(resource=resource)
    trace.set_tracer_provider(tracer_provider)
    collector_endpoint = f"http://{get_env_host()}:{get_env_port()}/v1/traces"
    span_exporter = OTLPSpanExporter(endpoint=collector_endpoint)
    simple_span_processor = SimpleSpanProcessor(span_exporter=span_exporter)
    trace.get_tracer_provider().add_span_processor(simple_span_processor)
    LlamaIndexInstrumentor().instrument(tracer_provider=tracer_provider)
    return trace.get_tracer(__name__)

# manual instrumentation 
def run():
    tracer = init_tracer()
    span = tracer.start_span(
        "apprun", attributes={SpanAttributes.OPENINFERENCE_SPAN_KIND: "CHAIN"}
    )
    with trace.use_span(span, end_on_exit=False):
        # configures global state so the right embedding models, LLMs will be used by the agent
        init_models()
        span.set_attribute(SpanAttributes.INPUT_VALUE, "")
        span.set_attribute(
            SpanAttributes.INPUT_MIME_TYPE, OpenInferenceMimeTypeValues.TEXT.value
        )
        # creates all the tools to be used by the agent
        tools = create_all_tools()

        output = agent2.run(tools)
        span.set_attribute(SpanAttributes.OUTPUT_VALUE, json.dumps(output.response))
        span.set_attribute(
            SpanAttributes.OUTPUT_MIME_TYPE, OpenInferenceMimeTypeValues.TEXT.value
        )

        span.end()

what i would really like is for "apprun" to be a top level trace with all other spans underneath it

RunLLM
APP
·
To ensure your manually created spans appear in the Phoenix dashboard alongside the llama-index spans, you need to ensure that the spans are correctly associated with the same trace context. Here are a few steps to troubleshoot and ensure proper integration: 1. Ensure Proper Context Propagation: Make sure that the context is correctly propagated across your application. This involves setting the context for the manually created spans and ensuring they are part of the same trace.2. Check Span Attributes: Verify that the span attributes are correctly set, especially the SpanAttributes.OPENINFERENCE_SPAN_KIND and other relevant attributes.3. Verify Tracer Initialization: Ensure that the tracer is correctly initialized and that the LlamaIndexInstrumentor is properly instrumented with the same tracer provider.Here's an updated version of your code with additional context propagation: ```from openinference.semconv.resource import ResourceAttributes from openinference.instrumentation.llama_index import LlamaIndexInstrumentor from opentelemetry import trace from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.resources import Resource from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import SimpleSpanProcessor from openinference.semconv.trace import SpanAttributes, OpenInferenceMimeTypeValues import jsondef init_tracer(spawn_server: bool = True): resource = Resource(attributes={ResourceAttributes.PROJECT_NAME: "viv4"}) tracer_provider = TracerProvider(resource=resource) trace.set_tracer_provider(tracer_provider) collector_endpoint = f"http://{get_env_host()}:{get_env_port()}/v1/traces" span_exporter = OTLPSpanExporter(endpoint=collector_endpoint) simple_span_processor = SimpleSpanProcessor(span_exporter=span_exporter) trace.get_tracer_provider().add_span_processor(simple_span_processor) LlamaIndexInstrumentor().instrument(tracer_provider=tracer_provider) return trace.get_tracer(name)def run(): tracer = init_tracer() with tracer.start_as_current_span("apprun", attributes={SpanAttributes.OPENINFERENCE_SPAN_KIND: "CHAIN"}) as span: span.set_attribute(SpanAttributes.INPUT_VALUE, "") span.set_attribute(SpanAttributes.INPUT_MIME_TYPE, OpenInferenceMimeTypeValues.TEXT.value)
# configures global state so the right embedding models, LLMs will be used by the agent init_models() # creates all the tools to be used by the agent tools = create_all_tools() output = agent2.run(tools) span.set_attribute(SpanAttributes.OUTPUT_VALUE, json.dumps(output.response)) span.set_attribute(SpanAttributes.OUTPUT_MIME_TYPE, OpenInferenceMimeTypeValues.TEXT.value)LBKS# Ensure the span is ended after the context block
run()```
RunLLM
APP
·
This code ensures that the apprun span is the top-level span and that all other spans are nested under it. The start_as_current_span method is used to ensure proper context propagation.Would you like more detail on any specific part?Sources: - Manual Instrumentation Guide
💡 Hint: Mention RunLLM in the thread for followups.
James C.
·
ah! it was an issue on my end, i was foolishly spawning the px server in the same process as my agent (that currently exits once it is "done") -- causes a race condition with processing the last span. having a persistent phoenix server fixed this. cheers!
👍1
Mikyo
·
Oh awesome James!