Hi there! I'm experiencing performance issues with Phoenix. As the number of traces grows, opening a project becomes slower and slower. Do you have a recommendation for mitigating these issues?
To mitigate performance issues in Phoenix as the number of traces grows, consider implementing lazy loading of spans and reducing the page size from 100 to 50 spans. This approach has been shown to help reduce load times significantly. Additionally, ensure you are using the latest version of Phoenix, as recent updates have included performance improvements for handling large datasets.Would you like more details on how to implement these improvements?Sources: - GitHub Issue #4006 - Slack Discussion
馃挕聽Hint: Mention RunLLM in the thread for followups.
hey Elliot P., thanks for reporting. can you tell us how you're using phoenix (self hosted, app.arize.com, etc.) and what version if you're self-hosting?
Hi Xander! We're self-hosting Phoenix in Kubernetes. The database is Postgres in Amazon Aurora with db.r7g.xlarge instances. Currently, we're on Phoenix 7.9.0.
Or if you have particular long-context LLM spans.
Just to clarify, the issue happens when you click into a particular project to view traces, not when you hit the projects (plural) page?
We've been deleting traces weekly to mitigate the performance issues. The number of traces each week isn't large (on the order of 10s of thousands), but some traces have a large number of spans (>100), which may themselves be significant (e.g. 5,000 prompt tokens).
Just to clarify, the issue happens when you click into a particular project to view traces, not when you hit the projects (plural) page?
Correct. /projects loads quickly. Clicking an individual project takes time (and that time is relative to the number of traces). Interestingly, clicking out and back into the project is very fast. Is it possible that Phoenix is loading all spans when the project page loads? Would it be feasible to only load the spans when one clicks a particular trace?
Correct. The slowness appears when hitting the traces list (e.g. /projects/UHJvamVjdDoy). Opening a particular trace (e.g. /projects/UHJvamVjdDoy/traces/f5a87692f4c96c061335dc94da5f36b9) is fast.
It sounds like you've noticed that deleting old traces from the project helps the performance of this page?
Yes, that's correct. We notice an immediate performance boost after running a command like DELETE FROM traces WHERE end_time < ...
Got it, thanks!
Thank you so much for the help, Xander. I will keep an eye on that ticket. Please let me know if you need any additional information from me.
