Yaozhong K.

In a multi-tenant LLM application, data isolation is your responsibility as the application builder, not the model or library provider’s. Caching is an architectural decision, and any caching layer (proxy-level like LiteLLM, or implicit provider-side caching behind a shared API key) must be designed with tenant boundaries in mind. If responses are cached purely on prompt similarity (or prompt hash) without tenant/user scoping, then the scenario you described is real.

Thanks! I’ll seek for help here if I run into any blockers. At some point I’d also love to chat with the team to see if there are anything we can do to support the community back too.