Phoenix 12.35.0 Email in Use Error with Keycloak Authentication on Kubernetes: Causes and Solutions

·Apr 20, 2026 02:17 PM·

Hello, I am running Phoenix 12.35.0 on Kubernetes with keycloak auth and occassionally run into the below error "Email in Use". It clears after I remove the user from the DB or from the users section in the UI. Any idea why this happens? It reoccurs again after a while and I need to repeat this process

9 comments

· Sorted by Oldest

RunLLM
APP
·
·
The "Email in Use" error occurs because Phoenix enforces a uniqueness constraint on user email addresses. If a user is deleted, but their database record is only soft-deleted (i.e., marked with a deleted_at timestamp), the email remains reserved and cannot be reused for new accounts. This is especially problematic with OAuth2/Keycloak, as users cannot re-register with the same email until the record is fully removed from the database. The issue is resolved temporarily when you manually delete the user record, but it will recur if the deletion process only marks the user as deleted instead of actually removing the record from the database. Recent Phoenix versions have addressed this by fully deleting user tuples instead of soft-deleting them, which resolves the email reuse problem for OAuth2 and SSO integrations. See the discussion and resolution in <https://github.com/Arize-ai/phoenix/issues/4676|this issue>. If you are still experiencing this issue on 12.35.0, it may indicate that your deployment is not applying the latest database migration or that some users are still being soft-deleted. Ensure your database schema is up to date and that user deletions are fully removing records, not just marking them as deleted. If the problem persists, check for any customizations or older migration scripts that might be interfering with the deletion logic. Would you like more detail on how to verify or update your database schema and user deletion behavior? Sources: - <https://github.com/Arize-ai/phoenix/issues/4676|https://github.com/Arize-ai/phoenix/issues/4676>
💡 Hint: Mention RunLLM in the thread for followups.
Xander S.
·
·
Hi Amith, what action are you taking that this error occurs? Does it happen when you attempt to log in?
Amith
·
·
Xander S. Yes, this happens while trying to login using keycloak. It sends me to keycloak page for login and on returning to Phoenix shows me this error.
Xander S.
·
·
The error you report in the your message differs from the one in the screenshot
Xander S.
·
·
It sounds like this consistently happens rather than being a one-off issue?
Roger Y.
·
·
That error means Phoenix’s OAuth callback got a (client_id, sub) pair it doesn’t recognize, but the email in the token already belongs to an existing Phoenix user tied to a different sub. Deleting the row clears the email collision; next login creates a fresh row with the current sub. So something on the Keycloak side is causing the sub (Keycloak’s internal user UUID) to change for that person over time. Before you delete the row next time, can you grab:
SELECT email, oauth2_client_id, oauth2_user_id, updated_at FROM users WHERE email = '<affected email>';
If oauth2_user_id (or oauth2_client_id) is different on each recurrence, that confirms it.
Amith
·
·
Xander S. Sorry, that was the message included in the app url on browser on redirect - The actual error is the one in the image. Roger Y. Understood, I will verify if this value is changing next time it happens. Thanks for your inputs.
1
Amith
·
·
Roger Y. It seems like we have identified the issue : the oath2_user_id value is changed when the issue occurs for a user. But we are not able to set this as a fixed value on keycloak side, sometimes when the pods are restarted the value seems to change. Any suggestions on how to fix this?
Roger Y.
·
(edited)·
Thanks for confirming. Keycloak’s sub should already be stable for a given user. If it is changing around pod restarts, you can check the folowing. 1. Persistence first. Make sure Keycloak’s DB is actually surviving restarts. If the pod is using ephemeral storage, or the external DB / PVC is not really persistent, users can get recreated with new internal UUIDs. 2. Startup import / reconcile path. Check whether anything on startup is recreating users: kc.sh import --override, partial import, Helm/operator bootstrap jobs, init containers, or other automation. --import-realm by itself should normally skip an existing realm, so I’d look more for overwrite / reconcile behavior than --import-realm alone. 3. LDAP/AD federation + Keycloak version. If these are LDAP/AD-backed users and you’re on Keycloak 24.0.1 or 24.0.2, keycloak#28248 is worth checking. That regression was fixed in 24.0.3 and could recreate federated users on certain update paths. 4. Stable federation ID. If you are using LDAP/AD federation, confirm the provider is using a stable unique attribute such as entryUUID (OpenLDAP) or objectGUID (AD), and that the federation settings are the same after restart.

RunLLM
APP
·
·
The "Email in Use" error occurs because Phoenix enforces a uniqueness constraint on user email addresses. If a user is deleted, but their database record is only soft-deleted (i.e., marked with a deleted_at timestamp), the email remains reserved and cannot be reused for new accounts. This is especially problematic with OAuth2/Keycloak, as users cannot re-register with the same email until the record is fully removed from the database. The issue is resolved temporarily when you manually delete the user record, but it will recur if the deletion process only marks the user as deleted instead of actually removing the record from the database. Recent Phoenix versions have addressed this by fully deleting user tuples instead of soft-deleting them, which resolves the email reuse problem for OAuth2 and SSO integrations. See the discussion and resolution in <https://github.com/Arize-ai/phoenix/issues/4676|this issue>. If you are still experiencing this issue on 12.35.0, it may indicate that your deployment is not applying the latest database migration or that some users are still being soft-deleted. Ensure your database schema is up to date and that user deletions are fully removing records, not just marking them as deleted. If the problem persists, check for any customizations or older migration scripts that might be interfering with the deletion logic. Would you like more detail on how to verify or update your database schema and user deletion behavior? Sources: - <https://github.com/Arize-ai/phoenix/issues/4676|https://github.com/Arize-ai/phoenix/issues/4676>
💡 Hint: Mention RunLLM in the thread for followups.
Xander S.
·
·
Hi Amith, what action are you taking that this error occurs? Does it happen when you attempt to log in?
Amith
·
·
Xander S. Yes, this happens while trying to login using keycloak. It sends me to keycloak page for login and on returning to Phoenix shows me this error.
Xander S.
·
·
The error you report in the your message differs from the one in the screenshot
Xander S.
·
·
It sounds like this consistently happens rather than being a one-off issue?
Roger Y.
·
·
That error means Phoenix’s OAuth callback got a (client_id, sub) pair it doesn’t recognize, but the email in the token already belongs to an existing Phoenix user tied to a different sub. Deleting the row clears the email collision; next login creates a fresh row with the current sub. So something on the Keycloak side is causing the sub (Keycloak’s internal user UUID) to change for that person over time. Before you delete the row next time, can you grab:
SELECT email, oauth2_client_id, oauth2_user_id, updated_at FROM users WHERE email = '<affected email>';
If oauth2_user_id (or oauth2_client_id) is different on each recurrence, that confirms it.
Amith
·
·
Xander S. Sorry, that was the message included in the app url on browser on redirect - The actual error is the one in the image. Roger Y. Understood, I will verify if this value is changing next time it happens. Thanks for your inputs.
1
Amith
·
·
Roger Y. It seems like we have identified the issue : the oath2_user_id value is changed when the issue occurs for a user. But we are not able to set this as a fixed value on keycloak side, sometimes when the pods are restarted the value seems to change. Any suggestions on how to fix this?
Roger Y.
·
(edited)·
Thanks for confirming. Keycloak’s sub should already be stable for a given user. If it is changing around pod restarts, you can check the folowing. 1. Persistence first. Make sure Keycloak’s DB is actually surviving restarts. If the pod is using ephemeral storage, or the external DB / PVC is not really persistent, users can get recreated with new internal UUIDs. 2. Startup import / reconcile path. Check whether anything on startup is recreating users: kc.sh import --override, partial import, Helm/operator bootstrap jobs, init containers, or other automation. --import-realm by itself should normally skip an existing realm, so I’d look more for overwrite / reconcile behavior than --import-realm alone. 3. LDAP/AD federation + Keycloak version. If these are LDAP/AD-backed users and you’re on Keycloak 24.0.1 or 24.0.2, keycloak#28248 is worth checking. That regression was fixed in 24.0.3 and could recreate federated users on certain update paths. 4. Stable federation ID. If you are using LDAP/AD federation, confirm the provider is using a stable unique attribute such as entryUUID (OpenLDAP) or objectGUID (AD), and that the federation settings are the same after restart.