Error Deploying Phoenix with Nginx: Bad Request to GraphQL

BLT · 2025-05-02T01:55:56.758Z

I am hitting an error when deploying phoenix behind nginx. The error is bad request to graphql Unexpected token ‘U’, unable to … is not a valid JSON. When I check the request payload, the query parameter is “…fragment APIKeysTableFragment on User {\n apiKeys…” For context, my nginx configuration location /phoenix_root_path/ { proxy_pass http://<>:6006/; proxy_set_header Host \$host; sub_filter ‘href=“/assets/‘ ‘href=“/phoenix_root_path/assets/‘ sub_filter ‘href=“/‘ ‘href=“/phoenix_root_path/‘; sub_filter_once off; In my helm chart, I have set PHOENIX_HOST_ROOT_PATH: /phoenix_root_path And PHOENIX_PROJECT_NAME and PHOENIX_HOST_URL both to be accessed by env var in my code.

14 comments

· Sorted by Oldest

RunLLM
APP
·
The error "Unexpected token ‘U’, ... is not valid JSON" when deploying Phoenix behind nginx with a path prefix is typically caused by a misconfiguration of the root path or reverse proxy, resulting in static assets or API endpoints being requested from the wrong location. The environment variable PHOENIX_HOST_ROOT_PATH must be set (as you did), but it only works correctly if the reverse proxy is configured to forward requests with the prefix and rewrite paths accordingly. The sub_filter directives alone are not sufficient; you must ensure nginx forwards the correct path and does not strip or double the prefix, and that all static assets and API calls are routed through /phoenix_root_path as expected by Phoenix. This is a common issue and is documented in the reverse proxy example and configuration docs.Additionally, the error can occur if the proxy is not rewriting paths correctly, causing the Phoenix server to return an HTML error page (e.g., 404 or 400) instead of JSON, which the frontend then tries to parse as JSON. Double-check your nginx config to ensure the location block matches the prefix, and that proxy_pass does not add or remove slashes incorrectly. See this issue and Slack discussion for similar troubleshooting. Would you like a step-by-step nginx config example for Phoenix behind a path prefix?Sources: - Phoenix reverse proxy example - GitHub Issue: static files fail to load with ingress prefix - configuration docs - Slack discussion
💡 Hint: Mention RunLLM in the thread for followups.
BLT
·
The error happens with graphql not able to parse payload
Xander S.
·
Hey BLT, can you upgrade to arize-phoenix: v8.31.0?
BLT
·
Ok will try. I have currently installed 8.26.3. Isit a known issue for ver below 8.31.0?
Xander S.
·
Yes
BLT
·
Alright thanks Xander S. will try!
🙌1
BLT
·
Hi Xander S. same error with graphql after upgrading to 8.31.0
Xander S.
·
Can you check the network tab to see what route is getting hit? I expect /phoenix_root_path/graphql
Xander S.
·
I don't think we have a PHOENIX_HOST_URL env var.
Xander S.
·
Not sure I understand the specific carve-out for the /assets directory. Is that not handled by the following line?
BLT
·
PHOENIX_HOST_URL is used as an env variable for my app to point to the endpoint. Shouldn’t have conflict
👍1
BLT
·
It’s weird I think could be my nginx setup. I have two separate pods for phoenix which will fail. If I have just one phoenix pod it works.
Xander S.
·
Worth also performing a sanity check to make sure the image has successfully updated. The following button was added in v8.30.0 https://github.com/Arize-ai/phoenix/pull/7358
Xander S.
·
Ah, I guess you can't check that without seeing the UI first

RunLLM
APP
·
The error "Unexpected token ‘U’, ... is not valid JSON" when deploying Phoenix behind nginx with a path prefix is typically caused by a misconfiguration of the root path or reverse proxy, resulting in static assets or API endpoints being requested from the wrong location. The environment variable PHOENIX_HOST_ROOT_PATH must be set (as you did), but it only works correctly if the reverse proxy is configured to forward requests with the prefix and rewrite paths accordingly. The sub_filter directives alone are not sufficient; you must ensure nginx forwards the correct path and does not strip or double the prefix, and that all static assets and API calls are routed through /phoenix_root_path as expected by Phoenix. This is a common issue and is documented in the reverse proxy example and configuration docs.Additionally, the error can occur if the proxy is not rewriting paths correctly, causing the Phoenix server to return an HTML error page (e.g., 404 or 400) instead of JSON, which the frontend then tries to parse as JSON. Double-check your nginx config to ensure the location block matches the prefix, and that proxy_pass does not add or remove slashes incorrectly. See this issue and Slack discussion for similar troubleshooting. Would you like a step-by-step nginx config example for Phoenix behind a path prefix?Sources: - Phoenix reverse proxy example - GitHub Issue: static files fail to load with ingress prefix - configuration docs - Slack discussion
💡 Hint: Mention RunLLM in the thread for followups.
BLT
·
The error happens with graphql not able to parse payload
Xander S.
·
Hey BLT, can you upgrade to arize-phoenix: v8.31.0?
BLT
·
Ok will try. I have currently installed 8.26.3. Isit a known issue for ver below 8.31.0?
Xander S.
·
Yes
BLT
·
Alright thanks Xander S. will try!
🙌1
BLT
·
Hi Xander S. same error with graphql after upgrading to 8.31.0
Xander S.
·
Can you check the network tab to see what route is getting hit? I expect /phoenix_root_path/graphql
Xander S.
·
I don't think we have a PHOENIX_HOST_URL env var.
Xander S.
·
Not sure I understand the specific carve-out for the /assets directory. Is that not handled by the following line?
BLT
·
PHOENIX_HOST_URL is used as an env variable for my app to point to the endpoint. Shouldn’t have conflict
👍1
BLT
·
It’s weird I think could be my nginx setup. I have two separate pods for phoenix which will fail. If I have just one phoenix pod it works.
Xander S.
·
Worth also performing a sanity check to make sure the image has successfully updated. The following button was added in v8.30.0 https://github.com/Arize-ai/phoenix/pull/7358
Xander S.
·
Ah, I guess you can't check that without seeing the UI first