Troubleshooting Deadlines Exceeded Errors in Self-Hosted Phoenix gRPC

RunLLM The connection is definitely configured correctly since only some spans are dropped due to this problem. I have tried HTTP and am still seeing dropped spans, but it manifests as a "remote closed connection" error

Eugene E. just for my understanding the traces are arriving but you start seeing timeouts? Do you have batching configured on the exporter? Any chance you can give us a rough estimate as to the volume of data you are sending?

Do you have batching configured on the exporter?

yes, i have tried it both ways (batch and simple) with grpc and http

Any chance you can give us a rough estimate as to the volume of data you are sending?

i'm trying to instrument llm eval runners - the current setup uses 8 parallel runners, each with its own exporter, so in the worst case it can submit 8 batches at a time. these are llm traces and traces instrumenting a complex modular agent workflow, there are llm traces at the bottom but the overall tree contains other, higher order logical nodes. these higher nodes can contain llm prompts as well, although i have already pruned almost all the fat from them and it didn't help. the collector is on a c4.8xlarge instance. i've tried running two collector containers in a docker compose - one only for writes and one for reads. both are backed by the same postgres

i'm seeing what look like timeouts on the exporter side and the traces don't look like they arrive in the collector

i don't see anything unusual in the collector logs although they are very basic and the phoenix docs don't seem to have a verbose log level config

So the traces never show up - is that correct? If so I wonder if it's networking or auth related?

most traces show up, about 80%

Oh I see. Hmm, let me do a bit more digging then. Thanks for the context. Appreciate it.

tvm

for some additional context, this is how i create things:

    tracer_provider = register(
        project_name=PHOENIX_APP_NAME,
        # protocol="http/protobuf",
        # protocol="grpc",
        auto_instrument=True,
        batch=True,
        set_global_tracer_provider=False,
    )
    PHOENIX_TRACER = tracer_provider.get_tracer(__name__)

i turned off set_global_tracer_provider because we have other tracing in other parts of our code that's not LLM-related and goes to DD, those tracers have their own provider and i wanted to keep them separate, not sure if this helps at all

to clarify - i've turned off the other tracing and the problem is still happening

That is helpful. For things like DD export are you running an OTEL agent? This might give you more nobs to process the spans before final export and offload the direct connection from your application to the phoenix instance. Another thing you could play with is the batch size. It might be that you are saturating the connections to Phoenix and need to batch even higher.

# export PHOENIX_COLLECTOR_ENDPOINT=http://localhost:6006

from opentelemetry import trace as trace_api
from phoenix.otel import TracerProvider, BatchSpanProcessor

tracer_provider = TracerProvider()
batch_processor = BatchSpanProcessor()
tracer_provider.add_span_processor(batch_processor)

https://opentelemetry.io/docs/specs/otel/trace/sdk/#batching-processor

maxQueueSize - the maximum queue size. After the size is reached spans are dropped. The default value is 2048.
scheduledDelayMillis - the maximum delay interval in milliseconds between two consecutive exports. The default value is 5000.
exportTimeoutMillis - how long the export can run before it is cancelled. The default value is 30000.
maxExportBatchSize - the maximum batch size of every export. It must be smaller or equal to maxQueueSize. If the queue reaches maxExportBatchSize a batch will be exported even if scheduledDelayMillis milliseconds have not elapsed. The default value is 512.

can i change these with env vars?

OTEL_BSP_MAX_QUEUE_SIZE
OTEL_BSP_SCHEDULE_DELAY
OTEL_BSP_MAX_EXPORT_BATCH_SIZE