Troubleshooting RESOURCE_EXHAUSTED Errors in Phoenix Tracing

The RESOURCE_EXHAUSTED error is due to the OpenTelemetry collector or Phoenix backend being unable to keep up with the rate of span ingestion, often because the batch processor queue fills up or the database (e.g., Postgres) is slow to insert spans. You can tune batch processor settings (batch size, queue size, export timeout) to help, but there is no built-in rate limiter or sampling configuration in Phoenix itself. To filter or sample spans, you can implement a custom SpanProcessor to drop or modify spans before export, as shown in the docs and examples below. Sampling every Nth trace is not natively supported, but can be implemented in a custom processor.To filter out certain spans, subclass BatchSpanProcessor and override on_start/on_end to skip spans matching your criteria (e.g., by name or attribute). Example: LBKSfrom phoenix.otel import register, BatchSpanProcessor from opentelemetry.context import Context from opentelemetry.sdk.trace import ReadableSpan, SpanLBKSclass FilteringSpanProcessor(BatchSpanProcessor): def _filter_condition(self, span: Span) -> bool: # Example: filter out spans named "secret_span" return span.name == "secret_span" def on_start(self, span: Span, parent_context: Context) -> None: if self._filter_condition(span): return super().on_start(span, parent_context) def on_end(self, span: ReadableSpan) -> None: if self._filter_condition(span): return super().on_end(span)LBKStracer_provider = register(batch=True) tracer_provider.add_span_processor(FilteringSpanProcessor(endpoint="http://localhost:6006/v1/traces", protocol="http/protobuf"))LBKSFor more, see Phoenix advanced span filtering and batch processor tuning. Would you like more detail on sampling or performance tuning?Sources: - Batch processor tuning and OOM issues - Phoenix advanced span filtering