Yeah we are looking into the query offloading - but will take a bit of time to narrow down the strategy. Def. do what you need to do!
LMK if you have time to catch up. We'll keep digging on our end as well.
https://calendar.app.google/SdrkCTheLESerL2G6
Good morning Dillon D. - we reviewed your issues last Friday and I don't think there's any easy fixes on our end. We've actually never gotten our Phoenix deployments that large. We're also working a bit blind as we don't quite know the state of your DB. It could be that there's a fair amount stored in TOAST (The Oversized-Attribute Storage Technique) and there are some heavy table scans going on.
The only immediate thing I think you could do is to start exporting to another phoenix instance as well. This way you can start treating your old phoenix as the cold tier.
would you be open to having a call with us? cc Roger Y.
Hey Dillon, sorry you're going through this. We'll do hard pass on stress testing to try to unblock you. Let me share your experience with the team before I take any more of your time.
Hey Dillon D. thanks for reaching out and sorry you are facing scaling issues. Appreciate you working with us.
There are a few things that are in the works to alleviate some of your issues:
we're introducing an explicit pre-upgrade migrations in our helm charts
Looks like there was a 1.0 release of starlette over the weekend that had some non backwards compatible signatures. We'll get a hotfix version out ASAP. sorry for the discruption.
https://github.com/Arize-ai/phoenix/pull/12332
No, they currently do not, one phoenix is considered one tenant. See https://arize.com/docs/phoenix/self-hosting/architecture#tenancy
There are some solutions for this in Phoenix documented on that page. We have some plans for this but not right at this second. For space level access control I'd consider Arize AX a good solution here.
