Mikyo

Hey Dillon just acking on my end. We will dig on. What attributes are you setting?

Hey Abrihinav, I'm not sure we have multimodal instrumented for nanobanana. Might check with the instrumentation team https://github.com/Arize-ai/openinference

Fix should be in the next release of cloud https://github.com/Arize-ai/phoenix/pull/12954

Hey Eric, thanks for reaching out. We can't repro so it might be something specific to your browser or setup that we're not able to understand. Would you mind filing a github issue with some of the details? Unfortunately we can't really repro the issue

Great to hear. Let us know if you need anything

Yeah we are looking into the query offloading - but will take a bit of time to narrow down the strategy. Def. do what you need to do! LMK if you have time to catch up. We'll keep digging on our end as well. https://calendar.app.google/SdrkCTheLESerL2G6

Good morning Dillon D. - we reviewed your issues last Friday and I don't think there's any easy fixes on our end. We've actually never gotten our Phoenix deployments that large. We're also working a bit blind as we don't quite know the state of your DB. It could be that there's a fair amount stored in TOAST (The Oversized-Attribute Storage Technique) and there are some heavy table scans going on. The only immediate thing I think you could do is to start exporting to another phoenix instance as well. This way you can start treating your old phoenix as the cold tier. would you be open to having a call with us? cc Roger Y.

Hey Dillon, sorry you're going through this. We'll do hard pass on stress testing to try to unblock you. Let me share your experience with the team before I take any more of your time.

Hey Dillon D. thanks for reaching out and sorry you are facing scaling issues. Appreciate you working with us. There are a few things that are in the works to alleviate some of your issues:

There are some limitations to the table vacuuming that might be affecting things (https://arize.com/docs/phoenix/settings/data-retention#why-doesn%E2%80%99t-my-storage-usage-decrease-after-deleting-traces). I'm inclined to believe the JSONB columns you have might not be getting cleaned up well here. What's the overall "non functional" behavior you are seeing? Happy to brainstorm some solutions to unblock you. AFK for a few hours. Will check back in.

awesome, thanks for being the canary in the coal mine