Troubleshooting Concurrent Trace Drops in Phoenix with PostgreSQL | Arize AI Community

Arize AI Community Icon

Troubleshooting Concurrent Trace Drops in Phoenix with PostgreSQL | Arize AI Community

shreeranga k.
·
RunLLM
RunLLM
APP
·
1. Settings for Concurrent Traces: Increasing the SQLALCHEMY_POOL_SIZE and SQLALCHEMY_MAX_OVERFLOW can help manage more concurrent database connections, which is crucial when handling many traces. Ensure that your PostgreSQL instance can handle the increased connections. Additionally, consider increasing the memory allocated to each pod if they are underutilized.2. Known Issues: Dropped traces can occur due to network issues, insufficient resources, or misconfigured settings. It's important to ensure that your OpenTelemetry setup is correctly configured to handle the load. According to Phoenix documentation, ensure that the PHOENIX_COLLECTOR_ENDPOINT is correctly set and that your network can handle the traffic.3. Monitoring and Debugging: Use the PHOENIX_LOG_LEVEL and OTEL_LOG_LEVEL set to DEBUG to gather more detailed logs. This can help identify where traces are being dropped. Additionally, monitor your PostgreSQL logs for any connection issues or errors.4. Additional Configuration Tweaks: Consider adjusting the OTEL_BSP_MAX_QUEUE_SIZE and OTEL_BSP_SCHEDULE_DELAY to better suit your workload. A larger queue size and a longer delay can help manage bursts of trace data. Also, ensure that your network and infrastructure can handle the increased load from concurrent requests.Would you like more detail on any of these points?Sources: - Phoenix documentation
💡 Hint: Mention RunLLM in the thread for followups.
shreeranga k.
·
I have tried all of these ..its not helping
RunLLM
APP
·
In Phoenix, the choice between using a BatchSpanProcessor and a SimpleSpanProcessor can significantly impact performance, especially under high load. - BatchSpanProcessor: This processor is generally recommended for production environments. It processes spans in batches, which can reduce the overhead of exporting spans individually. This is particularly useful when dealing with a high volume of traces, as it can help manage network and processing load more efficiently. According to Phoenix documentation, using a BatchSpanProcessor is usually the right choice for applications to avoid losing spans and to optimize performance.- SimpleSpanProcessor: This processor exports spans immediately as they are created. While this can be useful for debugging or development environments where you want immediate feedback, it can introduce significant overhead in production, especially with high trace volumes. This is because each span is processed and exported individually, which can lead to increased network traffic and processing time.If you are experiencing dropped traces, switching to a BatchSpanProcessor might help by reducing the load on your system and allowing for more efficient processing of spans.Would you like more detail on configuring these processors?Sources: - Phoenix documentation
💡 Hint: Mention RunLLM in the thread for followups.
shreeranga k.
·
i am using BatchSpanProcessor grpc_exporter = OTLPSpanExporter(endpoint=otlp_exporter_enpoint, insecure=os.getenv(“OTEL_EXPORTER_OTLP_INSECURE”, True)) _tracer_provider.add_span_processor(BatchSpanProcessor(grpc_exporter)) yes i have 5 replicas, i did this in order to cater the traffic
Roger Y.
·
can you try calling the following to force flush the traces built up in the batch span processor? Given the 5s delay interval between exports (OTEL_BSP_SCHEDULE_DELAY value: "5000") it could be that the processor is terminated before flushing all its traces
tracer_provider.force_flush()
Roger Y.
·
you can also try SimpleProcess to see if it improves the situation
Roger Y.
·
fyi OTEL_BSP_SCHEDULE_DELAY is not a server config. it’s a config on the trace exporter on the client side (BSP = BatchSpanProcessor)
shreeranga k.
·
I will try this, thanks in advance
👍1
shreeranga k.
·
The tracer_provider.force_flush() worked but its not async friendly .. we cannot use it. 😑 is there any other solution. i tried setting this OTEL_EXPORTER_OTLP_INSECURE setting to 2000 ms but still no use. Roger Y. RunLLM
RunLLM
APP
·
The Phoenix's documentation I have ingested does not mention how to handle asynchronous environments when using tracer_provider.force_flush(). You might want to provide more details about the specific asynchronous framework or environment you are using, as this could help in finding a more suitable solution. If this doesn't help, please reach out to the Phoenix support team for further assistance.
💡 Hint: Mention RunLLM in the thread for followups.
Roger Y.
·
It could be that this is only needed at exit since that’s how the spans are lost. You can try this to see if it works
atexit.register(tracer_provider.force_flush)

·

it may also need to not shut everything down on exit in the beginning

from opentelemetry.sdk.trace import TracerProvider

tracer_provider = TracerProvider(shutdown_on_exit=False)

atexit.register(lambda: [tracer_provider.force_flush(), tracer_provider.shutdown()])

·

alternatively we can try to run it in a thread

import asyncio

await asyncio.get_running_loop().run_in_executor(None, tracer_provider.force_flush)