In the production guide, it mentions horizontal scaling which is good but not much other detail. Wondering if there is any gotchas with running more than 1 instance at a time? Assuming its safe but hoping there's a pre-existing answer to this
馃挕聽Hint: Mention RunLLM in the thread for followups.
To my knowledge, Phoenix is fully robust to horizontal scaling. Tagging 馃敀[private user] in case he has any insight here.
for instance our ingest performance does not scale well with node count due to backend limitations
and in some limited occasions you might even run into ingestion problems, though we believe those cases are rare
Ah ok interesting. We are having ingest problems where large volumes of spans are causing our instance to become unavailable. We see memory maxing out when this happens (4gb 2vcpu)
My assumption is the batches are queued in memory for processing and it isn't processing fast enough leading to this backup. But maybe the problem is elsewhere?
hey Peter K. this is an ongoing issue we鈥檙e investigating, currently spans can get backed up because ingest is performance limited by multiple mechanisms and we鈥檙e investigating ways to improve it. we recently added a limit on the ingestion queue to prevent the server from running out of memory in these cases, can you verify that your phoenix is on the latest version?
Ah yep that explains it. We updated today. We were not getting this error before
Interestingly we weren't filling up our memory before either though 馃
