Migrating Phoenix Deployment to Auth: Managing Downtime and API Key Changes

Swaraj R. · 2025-01-17T20:49:58.771Z

@RunLLM was curious about how i should migrate from a phoneix deployment without auth to a deployment with auth as per the documentation, i can only get a system api key after enabling auth env vars and restarting my deployment however i have production jobs running that are sending traces to these servers the production jobs also need to be updated to use the system api key (we have custom tracer set up so it's a code change, not env var change) is there a recommended way to enable the auth with a new system api key without facing downtime to send traces? Or should i do the migration in a low traffic time ?

Migrating Phoenix Deployment to Auth: Managing Downtime and API Key Changes | Arize AI Community

RunLLM
APP
·
Sure thing! I'm looking through the available data to answer your question. Hold on a second... ⏳
Mikyo
·
Ooof, I'm sorry about that Swaraj R. - You can't have any downtime I'm guessing...
Mikyo
·
I'm wondering if you could attach an auth enabled phoenix to the same DB...
Mikyo
·
I haven't tried it though so just a word of caution there.
Mikyo
·
We can also maybe make the secret be usable for the APIs and have a fix that way
Swaraj R.
·
no sweat !
I'm wondering if you could attach an auth enabled phoenix to the same DB...
to clarify how would this avoid missing traces?
Swaraj R.
·
We can also maybe make the secret be usable for the APIs and have a fix that way
by secret you mean this one?
Mikyo
·
I'm just thinking out loud here but you could bring up another phoenix on the same postgres, setup auth, and then once you have system keys setup just point to the new phoenix
Mikyo
·
yeah that one...
Swaraj R.
·
yeah i thought of that too but a bit more infra work that way the secret option would be good tbh that way i can set it in my production jobs even before enabling auth on phoenix assuming sending the secret even if auth is not enabled won't stop the traces from coming through
Swaraj R.
·
no rush on this , but good to have for now we can do it at 12 am when it's unlikely we'll have a lot of traffic
👍1
Mikyo
·
This is great feedback. Let me try to scope it with Xander S. - will follow up with a ticket for you to track
🙏1
Swaraj R.
·
Awesome thank you for the quick response! slight clarification on the current usage of the PHOENIX_SECRET apart from adding it to my env vars for phoenix deployment, currently is there any thing else i need to do with it? I wasnt' sure if that needs to be passes a JWT token when making api calls (likely not right?)
Mikyo
·
The PHOENIX_SECRET is just a long random string of your choosing that's used for signing. You can change it if you ever have some sort of emergency. The JWT tokens are signed and created by phoenix itself. Hopefully that makes sense.
🙏1
Swaraj R.
·
Yupp aligns with my understanding thanks!

16 comments