Hi Phoenix Support - im trying to deploy the phonix with the helm chart deployment in the kube. App is up and running in the pod but i see always the probes are failing and the endpoint is not accessible.
💡 Hint: Mention RunLLM in the thread for followups.
Jonathan B. Xander S. do you have any ideas?
when you get a chance if you can send us the kubectl get events, and kubectl get pods Ashwin G. this will help debug the situation
ttp-ai-observability:
Container ID: containerd://70a9294601bdf436beca59b90138da90b588f8e3ddc4374a8ad0cb2fba368603
Image: docker.ci.artifacts.xxxx.com/hub-docker-release-remote/arizephoenix/phoenix:version-11.2.0-nonroot
Image ID: docker.ci.artifacts.xxxx.com/hub-docker-release-remote/arizephoenix/phoenix@sha256:f29c922c8880e30dc1f490167259d8905d9fb359d821f15fa30c625ba6689859
Port: 8080/TCP
Host Port: 0/TCP
SeccompProfile: RuntimeDefault
State: Running
Started: Thu, 24 Jul 2025 13:00:44 -0700
Ready: True
Restart Count: 0
Limits:
cpu: 1
ephemeral-storage: 1500M
memory: 2Gi
Requests:
cpu: 500m
ephemeral-storage: 1G
memory: 1Gi
Liveness: http-get http://:8080/healthz delay=0s timeout=1s period=10s #success=1 #failure=4
Readiness: http-get http://:8080/healthz delay=0s timeout=1s period=10s #success=1 #failure=4
Startup: http-get http://:8080/healthz delay=0s timeout=1s period=10s #success=1 #failure=60
Environment Variables from:
ttp-ai-observability-configmap ConfigMap Optional: false
Environment:
PHOENIX_SECRET: <set to the key 'PHOENIX_SECRET' in secret 'phoenix-secret'> Optional: false
PHOENIX_ADMIN_SECRET: <set to the key 'PHOENIX_ADMIN_SECRET' in secret 'phoenix-secret'> Optional: false
PHOENIX_DEFAULT_ADMIN_INITIAL_PASSWORD: <set to the key 'PHOENIX_DEFAULT_ADMIN_INITIAL_PASSWORD' in secret 'phoenix-secret'> Optional: false
PHOENIX_SMTP_PASSWORD: <set to the key 'PHOENIX_SMTP_PASSWORD' in secret 'phoenix-secret'> Optional: false
ELASTIC_APM_SERVICE_NAME: product-ai_ttp-ai-observability
ELASTIC_APM_ENVIRONMENT: undefined
SRE_NAMESPACE: product-ai
SRE_APP_NAME: ttp-ai-observability
SRE_ARTIFACT: undefined
SRE_AGENT_ID: b031e21f-b21a-495d-817f-847b61a920ef
SRE_AGENT_DEPLOYMENT_VERSION: 77665ff4
SRE_BUILD_VERSION: undefined
SRE_STAGE_NAME: undefined
SRE_CLUSTER_ID: useast-dev-az-003
Hi Jonathan B. / Xander S. - here is the details of describe pod. it is running
i have customized the port to be 8080 in my kitt.yml
ok, are you able to get the logs of the pod
kubectl get logs <pod_name>yes the pod logs are all listening
could you show me the output of those logs, it should have the reason why its failing its readiness probe
WHERE pg_catalog.pg_class.relname = %(table_name)s::VARCHAR AND pg_catalog.pg_class.relkind = ANY (ARRAY[%(param_1)s::VARCHAR, %(param_2)s::VARCHAR, %(param_3)s::VARCHAR, %(param_4)s::VARCHAR, %(param_5)s::VARCHAR]) AND pg_catalog.pg_table_is_visible(pg_catalog.pg_class.oid) AND pg_catalog.pg_namespace.nspname != %(nspname_1)s::VARCHAR
2025-07-24 20:00:47,434 INFO sqlalchemy.engine.Engine [generated in 0.00020s] {'table_name': 'alembic_version', 'param_1': 'r', 'param_2': 'p', 'param_3': 'f', 'param_4': 'v', 'param_5': 'm', 'nspname_1': 'pg_catalog'}
2025-07-24 20:00:47,449 INFO sqlalchemy.engine.Engine SELECT alembic_version.version_num
FROM alembic_version
2025-07-24 20:00:47,449 INFO sqlalchemy.engine.Engine [generated in 0.00015s] {}
2025-07-24 20:00:47,462 INFO sqlalchemy.engine.Engine COMMIT
---------------------------
✅ Migrations completed in 0.163 seconds.
INFO: Started server process [1]
INFO: Waiting for application startup.
██████╗ ██╗ ██╗ ██████╗ ███████╗███╗ ██╗██╗██╗ ██╗
██╔══██╗██║ ██║██╔═══██╗██╔════╝████╗ ██║██║╚██╗██╔╝
██████╔╝███████║██║ ██║█████╗ ██╔██╗ ██║██║ ╚███╔╝
██╔═══╝ ██╔══██║██║ ██║██╔══╝ ██║╚██╗██║██║ ██╔██╗
██║ ██║ ██║╚██████╔╝███████╗██║ ╚████║██║██╔╝ ██╗
╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚══════╝╚═╝ ╚═══╝╚═╝╚═╝ ╚═╝ v11.2.0
| ⭐️⭐️⭐️ Support Open Source ⭐️⭐️⭐️
| ⭐️⭐️⭐️ Star on GitHub! ⭐️⭐️⭐️
| https://github.com/Arize-ai/phoenix
|
| 🌎 Join our Community 🌎
| https://arize-ai.slack.com/join/shared_invite/zt-2w57bhem8-hq24MB6u7yE_ZF_ilOYSBw#/shared-invite/email
|
| 📚 Documentation 📚
| https://arize.com/docs/phoenix
|
| 🚀 Phoenix Server 🚀
| Phoenix UI: http://localhost:8080
|
| Authentication: False
| Allowed Origins: ['']
| Log traces:
| - gRPC: http://localhost:4317
| - HTTP: http://localhost:8080/v1/traces
| Storage: postgresql://***:***@10.20.193.202:5432/ttpobservability
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
INFO: 127.0.0.6:51087 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:57115 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:41847 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:47681 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:41031 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:39713 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:52519 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:56901 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:50871 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:39893 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:50429 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:41451 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:37289 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:41779 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:37055 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:45033 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:42261 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:59993 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:52215 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:40269 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:52747 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:32987 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:37269 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:43399 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:57079 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:58161 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:48987 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:60341 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:53683 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:35609 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:43975 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:46309 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:46739 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:34783 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:58515 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:52371 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:49961 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:41173 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:49605 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:47611 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:41289 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:51223 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:59075 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:57263 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:51547 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:37571 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:41627 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:45673 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:59749 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:47785 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:35613 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:52039 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:33257 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:35707 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:34467 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:41149 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:54461 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:47601 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:53901 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:39433 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:40131 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:33315 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:59917 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:57395 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:57093 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:55467 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:37205 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:45651 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:49601 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:40277 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:50023 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:48477 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:44415 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:50595 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:57441 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:37179 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:38889 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:58505 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:56139 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:41181 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:57571 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:33863 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:50647 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:52007 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:59791 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:53073 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:54855 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:59995 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:42575 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:53071 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:56707 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:33665 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:60283 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:43249 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:50669 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:56661 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:53467 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:40877 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:41183 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:51475 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:45915 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:53329 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:56663 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:47747 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:40127 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:42949 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:35625 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:50447 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:37787 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:47993 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:41399 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:39201 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:52481 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:39547 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:39253 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:56967 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:40947 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:44359 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:39951 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:36915 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:56673 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:41563 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:36147 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:57635 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:45853 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:50677 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:60379 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:56655 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:51675 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:36539 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:46559 - "GET /healthz HTTP/1.1" 200 OK
this does look healthy to me, are you able to confirm if this Storage: postgresql://***:***@10.20.193.202:5432/ttpobservability is the right database location
this is all listening, yes it is
the next thing I would like to try is
kubectl port-forward <pod_name> 8080:8080and check if you see the portal
yes i have done this already and the portal is up and running in my local