Hi friends - I’m using phoenix serve to collect traces from my backend in production and store them in a remote postgres database. Works great! However, phoenix serve also spins up the UI and causes graphql to continuously poll postgres, which has blown up my postgres egress. I don’t need the UI running on my production server – I only need to collect the traces from the backend process and write them to postgres. Is there a way to configure Phoenix to run as OTLP “collector only” without running the UI? Or at least reduce the graphql polling rate or something?
There is currently no supported way to run Phoenix as a pure OTLP "collector only" without the UI; the UI and GraphQL API are always enabled when running phoenix serve. Disabling just the UI or GraphQL polling is not possible via configuration, though authentication and access control features are being developed to help secure deployments. The frequent GraphQL polling is due to the frontend checking for new data, and cannot be tuned or disabled independently at this time. This limitation is acknowledged by the Phoenix team and tracked as a feature request for future releases.Would you like more detail on possible workarounds or updates on this feature?Sources:
- Slack discussion: No way to disable UI or GraphQL API only
💡 Hint: Mention RunLLM in the thread for followups.
And if not possible as suggested by the bot, I would appreciate any creative workaround that come to mind 🙂 thanks
I was able to successfully use phoenix --no-ui serve to prevent serving the UI. However, I'm still seeing a ton of background queries (using a ton of DB egress) even in this "collector only" configuration. Reviving this thread with more details, if one of you folks would kindly take a look 🧵
The frequent query is:
select
phoenix.generative_models.name,
phoenix.generative_models.provider,
phoenix.generative_models.start_time,
phoenix.generative_models.name_pattern,
phoenix.generative_models.is_built_in,
phoenix.generative_models.created_at,
phoenix.generative_models.updated_at,
phoenix.generative_models.deleted_at,
phoenix.generative_models.id,
token_prices_1.model_id,
token_prices_1.token_type,
token_prices_1.is_prompt,
token_prices_1.base_rate,
token_prices_1.customization,
token_prices_1.id as id_1
from
phoenix.generative_models
left outer join phoenix.token_prices as token_prices_1 on phoenix.generative_models.id = token_prices_1.model_id
where
phoenix.generative_models.deleted_at is null
order by
phoenix.generative_models.namewhich seems to be generated by SQLAlchemy from this function in the source code here 🔗. Via pg_audit, I've observed over 100,000 queries in the last 7 days, indicating a query about every ~5 seconds, with 62,000,000 rows processed. To be clear, there are no instances of the UI open anywhere. Can anyone help me understand why phoenix is querying the model info and prices every 5 seconds on the server, and if there's a way to stop this?
oh, this is for caching the models because they’re needed for cost calculation during span ingestion
i think we can revisit the design given the new issue you are raising here (which wasn’t something we had considered)
would you mind filing a issue for us to track this?
basically we would have to be more thrifty when updating the cache of model pricing (on the other hand, the models still need to be updatable by users at any time)
one simple thing we can do is to add a filter for updated_at to limit the volume of data transfer
Ah, I see - thanks for explaining
thank you for bringing this to our attention. i’ll work on this as soon as i can
🙏 thanks Roger, I'll keep an eye on the issue - let me know if I can be of any help