Phoenix Instance Reboot Issues: Exploring Connection Pool Solutions

·Jun 13, 2025 07:21 PM

We've noticed we have to reboot our phoenix instance every few days. This has been happening for quite some time across many versions so it doesn't seem new. Essentially, the UI starts to break with connection pool issues. Oddly, tracing continues to work. Kind of would expect both to start failing but 🤷


asyncpg.exceptions.InvalidAuthorizationSpecificationError: no pg_hba.conf entry for host "x.x.x.x", user "postgres", database "postgres", no encryption
self.dbapi_connection = connection = pool._invoke_creator(self)

File "/phoenix/env/sqlalchemy/engine/create.py", line 646, in connect
return dialect.connect(*cargs, **cparams)

We are using AWS aurora. Wondering if we should be using pgbouncer or something

8 comments

· Sorted by Oldest

RunLLM
APP
·
It seems likely that Phoenix can experience connection pool exhaustion or stale connections when running against AWS Aurora, leading to UI errors while tracing may still function. This is a known issue, especially if the database closes idle connections or if the pool size is too small for your workload. Increasing the connection pool size (e.g., `pool_size=10, max_overflow=20`) and ensuring proper database connection handling may help, but there is no explicit recommendation in the data sources to use pgbouncer with Phoenix, and some users have reported issues with prepared statements when using pgbouncer in certain modes (<https://github.com/Arize-ai/phoenix/issues/6296|see here>). Additionally, errors like `no pg_hba.conf entry ... no encryption` suggest that SSL/TLS settings or Aurora's network policies may intermittently block connections, especially after idle periods (<https://arize-ai.slack.com/archives/C04R3GXC8HK/p1733534274.490479|see here>). Using the correct connection string with `sslmode=require` is important, but there have been reports that Phoenix may strip SSL parameters in some cases. There is no clear evidence that pgbouncer is officially supported or recommended for Phoenix, and using it may require additional configuration (e.g., session pooling mode). Would you like more detail on connection pool configuration or troubleshooting Aurora-specific issues? Sources: - <https://github.com/Arize-ai/phoenix/issues/6296|pgbouncer compatibility and issues> - <https://arize-ai.slack.com/archives/C04R3GXC8HK/p1733534274.490479|see here>
💡 Hint: Mention RunLLM in the thread for followups.
Dustin N.
·
hi Peter K. sorry you're running into this, Phoenix uses SqlAlchemy for our ORM layer which manages its own connection "pooling" and I'm not super clear how much pgbouncer will help here. Our default pooling strategy is to use a StaticPool with a single connection, which might explain why we see reconnection issues with some setups. While we don't currently expose connection-related configuration on Phoenix, do you think being able to modify any of these options might be useful? https://docs.sqlalchemy.org/en/20/core/pooling.html
Rusty
·
Peter K. is your database setup to require TLS? If yes, what option(s) are you setting on your PHOENIX_SQL_DATABASE_URL to require TLS?

Rusty

For me, setting that variable like this, resolved a very similar issue.

environ['PHOENIX_SQL_DATABASE_URL'] = f'postgresql+asyncpg://{environ["POSTGRES_USERNAME"]}:{environ["POSTGRES_PASSWORD"]}@{environ["POSTGRES_HOST"]}:{environ["POSTGRES_PORT"]}/{environ["POSTGRES_DATABASE"]}?sslmode=require'

Rusty
·
In my case the container would somehow manage to connect to the DB over TLS at first and then fail to reconnect later on (with that same "no encryption") error.
Peter K.
·
Ah we don't have that on there. Gonna give that a shot. Thanks!
Peter K.
·
If that doesn't work, I'll be back to say which configs might be helpful to expose. Got distracted once I started looking at the docs and never got back to it 🫠
Dustin N.
·
Thanks Peter, it’s been a long while since I’ve worked with Aurora so I’m pretty unfamiliar with anything special we might have to expose

RunLLM
APP
·
It seems likely that Phoenix can experience connection pool exhaustion or stale connections when running against AWS Aurora, leading to UI errors while tracing may still function. This is a known issue, especially if the database closes idle connections or if the pool size is too small for your workload. Increasing the connection pool size (e.g., `pool_size=10, max_overflow=20`) and ensuring proper database connection handling may help, but there is no explicit recommendation in the data sources to use pgbouncer with Phoenix, and some users have reported issues with prepared statements when using pgbouncer in certain modes (<https://github.com/Arize-ai/phoenix/issues/6296|see here>). Additionally, errors like `no pg_hba.conf entry ... no encryption` suggest that SSL/TLS settings or Aurora's network policies may intermittently block connections, especially after idle periods (<https://arize-ai.slack.com/archives/C04R3GXC8HK/p1733534274.490479|see here>). Using the correct connection string with `sslmode=require` is important, but there have been reports that Phoenix may strip SSL parameters in some cases. There is no clear evidence that pgbouncer is officially supported or recommended for Phoenix, and using it may require additional configuration (e.g., session pooling mode). Would you like more detail on connection pool configuration or troubleshooting Aurora-specific issues? Sources: - <https://github.com/Arize-ai/phoenix/issues/6296|pgbouncer compatibility and issues> - <https://arize-ai.slack.com/archives/C04R3GXC8HK/p1733534274.490479|see here>
💡 Hint: Mention RunLLM in the thread for followups.
Dustin N.
·
hi Peter K. sorry you're running into this, Phoenix uses SqlAlchemy for our ORM layer which manages its own connection "pooling" and I'm not super clear how much pgbouncer will help here. Our default pooling strategy is to use a StaticPool with a single connection, which might explain why we see reconnection issues with some setups. While we don't currently expose connection-related configuration on Phoenix, do you think being able to modify any of these options might be useful? https://docs.sqlalchemy.org/en/20/core/pooling.html
Rusty
·
Peter K. is your database setup to require TLS? If yes, what option(s) are you setting on your PHOENIX_SQL_DATABASE_URL to require TLS?

Rusty

For me, setting that variable like this, resolved a very similar issue.

environ['PHOENIX_SQL_DATABASE_URL'] = f'postgresql+asyncpg://{environ["POSTGRES_USERNAME"]}:{environ["POSTGRES_PASSWORD"]}@{environ["POSTGRES_HOST"]}:{environ["POSTGRES_PORT"]}/{environ["POSTGRES_DATABASE"]}?sslmode=require'

Rusty
·
In my case the container would somehow manage to connect to the DB over TLS at first and then fail to reconnect later on (with that same "no encryption") error.
Peter K.
·
Ah we don't have that on there. Gonna give that a shot. Thanks!
Peter K.
·
If that doesn't work, I'll be back to say which configs might be helpful to expose. Got distracted once I started looking at the docs and never got back to it 🫠
Dustin N.
·
Thanks Peter, it’s been a long while since I’ve worked with Aurora so I’m pretty unfamiliar with anything special we might have to expose