When upgrading Phoenix, if the PHOENIX_SECRET environment variable changes (is regenerated or overwritten), all existing API keys become invalid, even if they still appear in the UI. Generating new API keys resolves the issue because they are signed with the new secret. To prevent this in future upgrades, ensure that PHOENIX_SECRET is persisted and not changed between deployments鈥攎anage it with a persistent Kubernetes Secret or external secrets manager if using Helm or similar tools. This behavior is expected and documented in user reports and official guidance.
This issue is not related to API key expiration or database migrations, but specifically to the secret used for signing and validating API keys. Changing the secret invalidates all previously issued keys, requiring regeneration of new ones for continued access. For more, see the linked discussions and documentation below.
Would you like more detail on how to persist PHOENIX_SECRET or manage secrets during upgrades?
Sources:
- <https://github.com/Arize-ai/phoenix/issues/11295|GitHub Issue: API Keys invalid after updating>
- <https://github.com/Arize-ai/phoenix/issues/9341|GitHub Issue: Upgrade from 11.3 to 11.29 broke api key>
