Handling Database Migration Errors in a Distroless Phoenix Container

Kevin S. · 2025-05-30T22:35:04.361Z

I'm curious what the procedure is to handle database migration errors? The phoenix container is distroless, we can't shell in and run an alembic command. I'm really nervous about this breaking our production instance when someone upgrades the container and doesn't realize there are breaking migrations.

Handling Database Migration Errors in a Distroless Phoenix Container | Arize AI Community

Kevin S.
·
What would be ideal is a way to manually run the migration to an offline replica first
RunLLM
APP
·
Phoenix runs database migrations automatically at startup, but if a migration fails and the container is distroless (no shell access), you must clone the Phoenix repository locally, navigate to src/phoenix/db, set the PHOENIX_SQL_DATABASE_URL environment variable, and run Alembic commands (e.g., alembic upgrade head or alembic stamp head) from your local environment to manually resolve migration issues. This is the only supported way to recover from a dirty migration state when you can't shell into the container directly, as described in the official docs and repo instructions. For production, always pin the Phoenix version and test migrations in a staging environment before upgrading to avoid unexpected breaking changes or downtime. If you need a shell for debugging or manual migration inside a container, you can build and run the arizephoenix/phoenix:latest-debug image, which includes a shell, or modify the Dockerfile to use a debug base image. For persistent issues, reach out to the Arize community Slack for support. Would you like more detail on the manual migration steps or Docker debug image usage? Sources: - Phoenix DB README: Manual Migrations - Phoenix Issue: Migration Errors & Alembic CLI
💡 Hint: Mention RunLLM in the thread for followups.
Kevin S.
·
That's not really something we can tolerate in production
Kevin S.
·
Curious how migrations are tested before released?
Mikyo
·
Hey Kevin S. we test all the migrations in CI https://github.com/Arize-ai/phoenix/blob/67245e6312be164fde8e2bad6a2329b228cd0caf/tests/integration/db_migrations/test_up_and_down_migrations.py#L17
Kevin S.
·
Gotcha. Apologies, this is new territory for me -- our database reliability policy has always been - do not use schema migration tools 🙂
Kevin S.
·
because of this exact reason
Mikyo
·
It certainly is a tricky thing and we try to not do migrations often other than for net new functionality. Most migrations are entirely new tables. We also only do migration on major bumps to give a heads up and they are documented here: https://github.com/Arize-ai/phoenix/blob/main/MIGRATION.md Hope that alleviates some fears 💀
Mikyo
·
you could certainly attach a new phoenix version on a replica to test out the migration before you move it on to the primary. Not sure if that works.
Mikyo
·
also we do have debug builds so you can shell into those
Kevin S.
·
Yea, this definitely helps
Kevin S.
·
I am unable to fix this migration so far tho
Kevin S.
·
i tried alembic stamp head, alembic upgrade head
Xander S.
·
Kevin S. Which migration is failing for you?
Xander S.
·
The Phoenix version and full stacktrace when booting Phoenix would help.

27 comments