We've noticed we have to reboot our phoenix instance every few days. This has been happening for quite some time across many versions so it doesn't seem new. Essentially, the UI starts to break with connection pool issues. Oddly, tracing continues to work. Kind of would expect both to start failing but 馃し
asyncpg.exceptions.InvalidAuthorizationSpecificationError: no pg_hba.conf entry for host "x.x.x.x", user "postgres", database "postgres", no encryption
self.dbapi_connection = connection = pool._invoke_creator(self)
File "/phoenix/env/sqlalchemy/engine/create.py", line 646, in connect
return dialect.connect(*cargs, **cparams)We are using AWS aurora. Wondering if we should be using pgbouncer or something
It seems likely that Phoenix can experience connection pool exhaustion or stale connections when running against AWS Aurora, leading to UI errors while tracing may still function. This is a known issue, especially if the database closes idle connections or if the pool size is too small for your workload. Increasing the connection pool size (e.g., pool_size=10, max_overflow=20) and ensuring proper database connection handling may help, but there is no explicit recommendation in the data sources to use pgbouncer with Phoenix, and some users have reported issues with prepared statements when using pgbouncer in certain modes (see here).Additionally, errors like no pg_hba.conf entry ... no encryption suggest that SSL/TLS settings or Aurora's network policies may intermittently block connections, especially after idle periods (see here). Using the correct connection string with sslmode=require is important, but there have been reports that Phoenix may strip SSL parameters in some cases. There is no clear evidence that pgbouncer is officially supported or recommended for Phoenix, and using it may require additional configuration (e.g., session pooling mode). Would you like more detail on connection pool configuration or troubleshooting Aurora-specific issues?Sources:
- pgbouncer compatibility and issues
- see here
馃挕聽Hint: Mention RunLLM in the thread for followups.
hi Peter K. sorry you're running into this, Phoenix uses SqlAlchemy for our ORM layer which manages its own connection "pooling" and I'm not super clear how much pgbouncer will help here. Our default pooling strategy is to use a StaticPool with a single connection, which might explain why we see reconnection issues with some setups. While we don't currently expose connection-related configuration on Phoenix, do you think being able to modify any of these options might be useful? https://docs.sqlalchemy.org/en/20/core/pooling.html
For me, setting that variable like this, resolved a very similar issue.
environ['PHOENIX_SQL_DATABASE_URL'] = f'postgresql+asyncpg://{environ["POSTGRES_USERNAME"]}:{environ["POSTGRES_PASSWORD"]}@{environ["POSTGRES_HOST"]}:{environ["POSTGRES_PORT"]}/{environ["POSTGRES_DATABASE"]}?sslmode=require'In my case the container would somehow manage to connect to the DB over TLS at first and then fail to reconnect later on (with that same "no encryption") error.
Ah we don't have that on there. Gonna give that a shot. Thanks!
If that doesn't work, I'll be back to say which configs might be helpful to expose. Got distracted once I started looking at the docs and never got back to it 馃珷
Thanks Peter, it鈥檚 been a long while since I鈥檝e worked with Aurora so I鈥檓 pretty unfamiliar with anything special we might have to expose