Issues with Phoenix Traces in Azure Production App Deployment
Want to raise an issue we are seeing with our phoenix deployment (most likely not phoenix related but wanted to see if folks here have any ideas) Issue
Our production app has stopped sending phoenix traces though we have not changed anything
Context
Our set up is
a production app on Azure Function Apps
a phoenix server running on an Azure VM
Production app's main_app.py (entry point), instantiates the intrument() call
tracer_provider = trace_sdk.TracerProvider()
span_exporter = OTLPSpanExporter(
urljoin(os.environ["PHOENIX_COLLECTOR_ENDPOINT"], "/v1/traces")
)
span_processor = SimpleSpanProcessor(span_exporter)
tracer_provider.add_span_processor(span_processor)
trace_api.set_tracer_provider(tracer_provider)
LlamaIndexInstrumentor().instrument()The above runs whenever a new instance of a function app is invoked
Additionally we have custom top level traces that wrap each user request with an AGENT trace
with tracer.start_as_current_span("some_name"):
logging.info(f"Tracer Type: {type(tracer)}")
span = trace_api.get_current_span()
logging.info(f"Span Type: {type(span)}")
span.set_attribute("openinference.span.kind", "AGENT")We are on version 3.17.1 and have been for some time without any changes
Our llamaindex instrumentation version is openinference-instrumentation-llama-index==1.2.0
Some investigation
We are not seeing any traces coming from the production app
Spinning up the app and phoenix server locally works ✅
Spinning up the app locally and sending traces to production phoenix works ✅
Logging the Span Types in production for these top level traces are NonRecordingSpan type, locally these spans are _Span type
It's really weird that no spans are being exported anymore and was curious if any of you here have any ideas or have seen this before?
