Hi Phoenix Support - im trying to deploy the phonix with the helm chart deployment in the kube. App is up and running in the pod but i see always the probes are failing and the endpoint is not accessible.
If Phoenix is running in the pod but liveness/readiness probes are failing and the endpoint is inaccessible, check that the container ports (default: 6006 for HTTP, 4317 for gRPC) are correctly exposed in your deployment and service, and that the probes are configured to use the correct path and port. The Helm chart defaults to /healthz for liveness and /readyz for readiness on the server.port (default 6006) — ensure your values.yaml and service match these settings. Also, verify that your pod has permissions to bind to the port and that no security context or read-only filesystem is blocking startup or probe access. You can customize probe settings in your values.yaml under healthChecks.livenessProbe and healthChecks.readinessProbe, and adjust the port under server.port. For persistent issues, check pod logs for errors related to port binding or filesystem permissions, as these are common causes of probe failures in Kubernetes deployments of Phoenix. See Helm deployment docs and probe configuration for details.Would you like a step-by-step troubleshooting checklist or example values.yaml?Sources:
- Helm deployment.yaml (probe config)
- Helm values.yaml (probe settings)
- Helm deployment docs
💡 Hint: Mention RunLLM in the thread for followups.
Jonathan B. Xander S. do you have any ideas?
when you get a chance if you can send us the kubectl get events, and kubectl get pods Ashwin G. this will help debug the situation
ttp-ai-observability:
Container ID: containerd://70a9294601bdf436beca59b90138da90b588f8e3ddc4374a8ad0cb2fba368603
Image: docker.ci.artifacts.xxxx.com/hub-docker-release-remote/arizephoenix/phoenix:version-11.2.0-nonroot
Image ID: docker.ci.artifacts.xxxx.com/hub-docker-release-remote/arizephoenix/phoenix@sha256:f29c922c8880e30dc1f490167259d8905d9fb359d821f15fa30c625ba6689859
Port: 8080/TCP
Host Port: 0/TCP
SeccompProfile: RuntimeDefault
State: Running
Started: Thu, 24 Jul 2025 13:00:44 -0700
Ready: True
Restart Count: 0
Limits:
cpu: 1
ephemeral-storage: 1500M
memory: 2Gi
Requests:
cpu: 500m
ephemeral-storage: 1G
memory: 1Gi
Liveness: http-get http://:8080/healthz delay=0s timeout=1s period=10s #success=1 #failure=4
Readiness: http-get http://:8080/healthz delay=0s timeout=1s period=10s #success=1 #failure=4
Startup: http-get http://:8080/healthz delay=0s timeout=1s period=10s #success=1 #failure=60
Environment Variables from:
ttp-ai-observability-configmap ConfigMap Optional: false
Environment:
PHOENIX_SECRET: <set to the key 'PHOENIX_SECRET' in secret 'phoenix-secret'> Optional: false
PHOENIX_ADMIN_SECRET: <set to the key 'PHOENIX_ADMIN_SECRET' in secret 'phoenix-secret'> Optional: false
PHOENIX_DEFAULT_ADMIN_INITIAL_PASSWORD: <set to the key 'PHOENIX_DEFAULT_ADMIN_INITIAL_PASSWORD' in secret 'phoenix-secret'> Optional: false
PHOENIX_SMTP_PASSWORD: <set to the key 'PHOENIX_SMTP_PASSWORD' in secret 'phoenix-secret'> Optional: false
ELASTIC_APM_SERVICE_NAME: product-ai_ttp-ai-observability
ELASTIC_APM_ENVIRONMENT: undefined
SRE_NAMESPACE: product-ai
SRE_APP_NAME: ttp-ai-observability
SRE_ARTIFACT: undefined
SRE_AGENT_ID: b031e21f-b21a-495d-817f-847b61a920ef
SRE_AGENT_DEPLOYMENT_VERSION: 77665ff4
SRE_BUILD_VERSION: undefined
SRE_STAGE_NAME: undefined
SRE_CLUSTER_ID: useast-dev-az-003
Hi Jonathan B. / Xander S. - here is the details of describe pod. it is running
i have customized the port to be 8080 in my kitt.yml
ok, are you able to get the logs of the pod
kubectl get logs <pod_name>yes the pod logs are all listening
could you show me the output of those logs, it should have the reason why its failing its readiness probe
WHERE pg_catalog.pg_class.relname = %(table_name)s::VARCHAR AND pg_catalog.pg_class.relkind = ANY (ARRAY[%(param_1)s::VARCHAR, %(param_2)s::VARCHAR, %(param_3)s::VARCHAR, %(param_4)s::VARCHAR, %(param_5)s::VARCHAR]) AND pg_catalog.pg_table_is_visible(pg_catalog.pg_class.oid) AND pg_catalog.pg_namespace.nspname != %(nspname_1)s::VARCHAR
2025-07-24 20:00:47,434 INFO sqlalchemy.engine.Engine [generated in 0.00020s] {'table_name': 'alembic_version', 'param_1': 'r', 'param_2': 'p', 'param_3': 'f', 'param_4': 'v', 'param_5': 'm', 'nspname_1': 'pg_catalog'}
2025-07-24 20:00:47,449 INFO sqlalchemy.engine.Engine SELECT alembic_version.version_num
FROM alembic_version
2025-07-24 20:00:47,449 INFO sqlalchemy.engine.Engine [generated in 0.00015s] {}
2025-07-24 20:00:47,462 INFO sqlalchemy.engine.Engine COMMIT
---------------------------
✅ Migrations completed in 0.163 seconds.
INFO: Started server process [1]
INFO: Waiting for application startup.
██████╗ ██╗ ██╗ ██████╗ ███████╗███╗ ██╗██╗██╗ ██╗
██╔══██╗██║ ██║██╔═══██╗██╔════╝████╗ ██║██║╚██╗██╔╝
██████╔╝███████║██║ ██║█████╗ ██╔██╗ ██║██║ ╚███╔╝
██╔═══╝ ██╔══██║██║ ██║██╔══╝ ██║╚██╗██║██║ ██╔██╗
██║ ██║ ██║╚██████╔╝███████╗██║ ╚████║██║██╔╝ ██╗
╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚══════╝╚═╝ ╚═══╝╚═╝╚═╝ ╚═╝ v11.2.0
| ⭐️⭐️⭐️ Support Open Source ⭐️⭐️⭐️
| ⭐️⭐️⭐️ Star on GitHub! ⭐️⭐️⭐️
| https://github.com/Arize-ai/phoenix
|
| 🌎 Join our Community 🌎
| https://arize-ai.slack.com/join/shared_invite/zt-2w57bhem8-hq24MB6u7yE_ZF_ilOYSBw#/shared-invite/email
|
| 📚 Documentation 📚
| https://arize.com/docs/phoenix
|
| 🚀 Phoenix Server 🚀
| Phoenix UI: http://localhost:8080
|
| Authentication: False
| Allowed Origins: ['']
| Log traces:
| - gRPC: http://localhost:4317
| - HTTP: http://localhost:8080/v1/traces
| Storage: postgresql://***:***@10.20.193.202:5432/ttpobservability
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
INFO: 127.0.0.6:51087 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:57115 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:41847 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:47681 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:41031 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:39713 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:52519 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:56901 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:50871 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:39893 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:50429 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:41451 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:37289 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:41779 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:37055 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:45033 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:42261 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:59993 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:52215 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:40269 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:52747 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:32987 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:37269 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:43399 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:57079 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:58161 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:48987 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:60341 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:53683 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:35609 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:43975 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:46309 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:46739 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:34783 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:58515 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:52371 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:49961 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:41173 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:49605 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:47611 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:41289 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:51223 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:59075 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:57263 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:51547 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:37571 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:41627 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:45673 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:59749 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:47785 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:35613 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:52039 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:33257 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:35707 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:34467 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:41149 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:54461 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:47601 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:53901 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:39433 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:40131 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:33315 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:59917 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:57395 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:57093 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:55467 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:37205 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:45651 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:49601 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:40277 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:50023 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:48477 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:44415 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:50595 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:57441 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:37179 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:38889 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:58505 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:56139 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:41181 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:57571 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:33863 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:50647 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:52007 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:59791 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:53073 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:54855 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:59995 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:42575 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:53071 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:56707 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:33665 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:60283 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:43249 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:50669 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:56661 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:53467 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:40877 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:41183 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:51475 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:45915 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:53329 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:56663 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:47747 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:40127 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:42949 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:35625 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:50447 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:37787 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:47993 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:41399 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:39201 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:52481 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:39547 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:39253 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:56967 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:40947 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:44359 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:39951 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:36915 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:56673 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:41563 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:36147 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:57635 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:45853 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:50677 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:60379 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:56655 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:51675 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:36539 - "GET /healthz HTTP/1.1" 200 OK
INFO: 127.0.0.6:46559 - "GET /healthz HTTP/1.1" 200 OK
this does look healthy to me, are you able to confirm if this Storage: postgresql://***:***@10.20.193.202:5432/ttpobservability is the right database location
this is all listening, yes it is
the next thing I would like to try is
kubectl port-forward <pod_name> 8080:8080and check if you see the portal
yes i have done this already and the portal is up and running in my local