The RESOURCE_EXHAUSTED error is due to the OpenTelemetry collector or Phoenix backend being unable to keep up with the rate of span ingestion, often because the batch processor queue fills up or the database (e.g., Postgres) is slow to insert spans. You can tune batch processor settings (batch size, queue size, export timeout) to help, but there is no built-in rate limiter or sampling configuration in Phoenix itself. To filter or sample spans, you can implement a custom SpanProcessor to drop or modify spans before export, as shown in the docs and examples below. Sampling every Nth trace is not natively supported, but can be implemented in a custom processor.
To filter out certain spans, subclass BatchSpanProcessor and override on_start/on_end to skip spans matching your criteria (e.g., by name or attribute). Example:
```
from phoenix.otel import register, BatchSpanProcessor
from opentelemetry.context import Context
from opentelemetry.sdk.trace import ReadableSpan, Span
class FilteringSpanProcessor(BatchSpanProcessor):
def _filter_condition(self, span: Span) -> bool:
# Example: filter out spans named "secret_span"
return span.name == "secret_span"
def on_start(self, span: Span, parent_context: Context) -> None:
if self._filter_condition(span): return
super().on_start(span, parent_context)
def on_end(self, span: ReadableSpan) -> None:
if self._filter_condition(span): return
super().on_end(span)
tracer_provider = register(batch=True)
tracer_provider.add_span_processor(FilteringSpanProcessor(endpoint="http://localhost:6006/v1/traces", protocol="http/protobuf"))
```
For more, see <https://arize.com/docs/phoenix/tracing/how-to-tracing/advanced/modifying-spans|Phoenix advanced span filtering> and <https://github.com/Arize-ai/phoenix/issues/6869|batch processor tuning>. Would you like more detail on sampling or performance tuning?
Sources:
- <https://github.com/Arize-ai/phoenix/issues/9461|Batch processor tuning and OOM issues>
- <https://arize.com/docs/phoenix/tracing/how-to-tracing/advanced/modifying-spans|Phoenix advanced span filtering>
