Is there any recommended specs for the self host? We've got this in ecs with postgres but it just gets insanely slow after a day of use. We're doing about 30k traces a day. None of the resources look to be struggling but just wanna make sure we're in a good spot
It seems likely that for self-hosting Phoenix, the recommended specs can vary based on workload. Internally, Phoenix uses 2GB RAM and a single CPU for hosted instances, but this might not be sufficient for high workloads like 30k traces a day. You might need to scale up the number of instances or resources as you encounter performance issues. Additionally, Phoenix is stateless, so scaling out by deploying multiple containers backed by a single Postgres database is possible, which might help with performance issues (source).If you're experiencing slowness, it might also be related to the database configuration or the way data is being queried. For instance, performance issues have been reported when querying data beyond a certain time interval, and optimizations like lazy loading of spans and reducing page size have been suggested (source). Would you like more details on optimizing your setup or specific configurations? Sources: - https://arize-ai.slack.com/archives/C04R3GXC8HK/p1727401604.563959 - https://github.com/Arize-ai/phoenix/issues/6201
馃挕聽Hint: Mention RunLLM in the thread for followups.
Sorry for the delay getting back to you. This turned out to be PEBKAC 馃ぃ. Somehow we flipped back over to sqlite
