Improving Performance in Phoenix: Tips to Mitigate Slowness

Elliot P. · 2025-01-29T19:39:31.425Z

Hi there! I'm experiencing performance issues with Phoenix. As the number of traces grows, opening a project becomes slower and slower. Do you have a recommendation for mitigating these issues?

Improving Performance in Phoenix: Tips to Mitigate Slowness | Arize AI Community

RunLLM
APP
·
To mitigate performance issues in Phoenix as the number of traces grows, consider implementing lazy loading of spans and reducing the page size from 100 to 50 spans. This approach has been shown to help reduce load times significantly. Additionally, ensure you are using the latest version of Phoenix, as recent updates have included performance improvements for handling large datasets.Would you like more details on how to implement these improvements?Sources: - GitHub Issue #4006 - Slack Discussion
💡 Hint: Mention RunLLM in the thread for followups.
Xander S.
·
hey Elliot P., thanks for reporting. can you tell us how you're using phoenix (self hosted, app.arize.com, etc.) and what version if you're self-hosting?
Elliot P.
·
Hi Xander! We're self-hosting Phoenix in Kubernetes. The database is Postgres in Amazon Aurora with db.r7g.xlarge instances. Currently, we're on Phoenix 7.9.0.
Xander S.
·
Thanks Elliot P., super helpful to know. Can you tell us more about the nature and volume of your traces? For example, do you know the total amount of data in your DB?
Xander S.
·
Or if you have particular long-context LLM spans.
Xander S.
·
Just to clarify, the issue happens when you click into a particular project to view traces, not when you hit the projects (plural) page?
Elliot P.
·
We've been deleting traces weekly to mitigate the performance issues. The number of traces each week isn't large (on the order of 10s of thousands), but some traces have a large number of spans (>100), which may themselves be significant (e.g. 5,000 prompt tokens).
Elliot P.
·
Just to clarify, the issue happens when you click into a particular project to view traces, not when you hit the projects (plural) page?
Correct. /projects loads quickly. Clicking an individual project takes time (and that time is relative to the number of traces). Interestingly, clicking out and back into the project is very fast. Is it possible that Phoenix is loading all spans when the project page loads? Would it be feasible to only load the spans when one clicks a particular trace?
Xander S.
·
Hey Elliot P., thanks for clarifying. We do paginate the traces page, so we are not loading all of the traces at once. Just to clarify, you're hitting this slowness when you hit the traces tab, not the spans tab?
Elliot P.
·
Correct. The slowness appears when hitting the traces list (e.g. /projects/UHJvamVjdDoy). Opening a particular trace (e.g. /projects/UHJvamVjdDoy/traces/f5a87692f4c96c061335dc94da5f36b9) is fast.
Xander S.
·
Thanks Elliot P.. I've filed an issue here. It sounds like you've noticed that deleting old traces from the project helps the performance of this page? That is somewhat surprising given that we are paginating.
💯1
Elliot P.
·
It sounds like you've noticed that deleting old traces from the project helps the performance of this page?
Yes, that's correct. We notice an immediate performance boost after running a command like DELETE FROM traces WHERE end_time < ...
Xander S.
·
Got it, thanks!
Elliot P.
·
Thank you so much for the help, Xander. I will keep an eye on that ticket. Please let me know if you need any additional information from me.
Xander S.
·
Thanks so much Elliot P.!

21 comments