I wonder how phoenix works when the PHOENIX_COLLECTOR_ENDPOINT isn't configured properly or when the phoenix server isn't responding, it looks my application would still run, but just the traces won't show up. In such case, does phoenix simple drop the traces or keep accumulating traces in memory till out of memory?
It could be possible that if PHOENIX_COLLECTOR_ENDPOINT is not configured properly, your application will still run, but traces will not be sent to the Phoenix collector and thus won't show up in the UI. The Phoenix client or instrumentation will likely drop the traces if it cannot reach the configured endpoint, rather than accumulating them in memory indefinitely鈥攖here is no explicit mention in the data sources of Phoenix buffering traces until out of memory in this scenario. This behavior is consistent with typical OpenTelemetry exporters, which may retry a few times but eventually drop data if the collector is unreachable (source, source).Though I'm not entirely certain, it might be that if you are running Phoenix in a notebook or with launch_app, traces are kept in a temporary session and lost when the session ends, but not accumulated indefinitely in memory. For persistent deployments (e.g., running phoenix.server.main serve), traces are written to disk (SQLite/Postgres) if configured, but if the endpoint is misconfigured, traces simply won't be collected (source, source). Would you like more detail on the specific failure modes or error handling?Sources:
- https://github.com/Arize-ai/phoenix/issues/1487
- https://github.com/Arize-ai/phoenix/issues/4921
- https://arize-ai.slack.com/archives/C04R3GXC8HK/p1722534335.071179
- https://docs.arize.com/phoenix/self-hosting/configuration#client-configuration
馃挕聽Hint: Mention RunLLM in the thread for followups.
