Troubleshooting 404 Error on Phoenix App Deployed in AWS ECS Fargate

Chaitanya · 2024-12-16T17:27:13.088Z

Hi All. I am facing this weird issue. I am using the latest docker version of phoenix. I pushed it to internal ECR in AWS and deployed on ECS fargate, with PHOENIX_HOST_ROOT_PATH as /phoenix/v1 . Container is created successfully and I can access the logs that the UI is up at the given path. I created a DNS record also for this so that I can access the UI from dev environment. When I try to access the website , it gives 404 not found error. When I try with /phoenix/v1/ , I get an HTML page with below errors. Its an empty white page. How to fix this issue?

Troubleshooting 404 Error on Phoenix App Deployed in AWS ECS Fargate | Arize AI Community

10 comments

· Sorted by Oldest

RunLLM
APP
·
The issue you're experiencing is likely related to the configuration of the PHOENIX_HOST_ROOT_PATH environment variable and the setup of your reverse proxy. According to this GitHub issue, the PHOENIX_HOST_ROOT_PATH is intended to work with a reverse proxy. If the proxy is not configured correctly to handle the path prefix, it can result in 404 errors or a blank page. Ensure that your reverse proxy is correctly set up to handle the /phoenix/v1 path. You might need to adjust the proxy settings to rewrite the URL paths appropriately. Additionally, verify that the static assets are being served from the correct path. If you need more detailed guidance on setting up a reverse proxy, let me know.Sources: - GitHub Issue 3961
💡 Hint: Mention RunLLM in the thread for followups.
Chaitanya
·
Also there is a health check I have added for the ECS. It will ping the /healthz endpoint periodically. Actually it should ping /phoenix/v1/healthz but as per below log, its different. INFO: 10.41.248.21:63560 - "GET /phoenix/v1/phoenix/v1/healthz HTTP/1.1" 200 OK
Roger Y.
·
Since the health check is returning a 200 status code, it seems to be functioning as expected and may not be the root cause of the issue. Proxy-related challenges can be complex to diagnose externally, as they often depend on specific vendor configurations. Here’s a link to a similar situation that others have encountered, which might provide some helpful context.
Chaitanya
·
Roger Y. do you want me to try with the debug image here?
Roger Y.
·
are you already using the latest version? in that case you may not need the debug version
Chaitanya
·
Yeah I am using the latest version only. I am using the ALB in AWS. How can I debug the error?
Roger Y.
·
I’m not entirely sure, as I don’t have the infrastructure to replicate the issue. However, could you take a look in your browser and check the URL for modernizr.js as shown in the screenshot below? Would you be able to paste the URL into your browser and experiment with it until it returns the actual file?
Chaitanya
·
I found the file. There is an error "Unexpected token <"
Roger Y.
·
That’s not the actual file, because the html is returned by the load balancer when it’s not found. The actual file should be a text file like below
Chaitanya
·
So I removed the path-patterns in the listener rule configurations and the environment variable for host root path. After that I can see the UI and API are both working. It will be better to have an example for deploying the phoenix in any cloud platform with various settings for paths. Thanks Roger .
🙌1

RunLLM
APP
·
The issue you're experiencing is likely related to the configuration of the PHOENIX_HOST_ROOT_PATH environment variable and the setup of your reverse proxy. According to this GitHub issue, the PHOENIX_HOST_ROOT_PATH is intended to work with a reverse proxy. If the proxy is not configured correctly to handle the path prefix, it can result in 404 errors or a blank page. Ensure that your reverse proxy is correctly set up to handle the /phoenix/v1 path. You might need to adjust the proxy settings to rewrite the URL paths appropriately. Additionally, verify that the static assets are being served from the correct path. If you need more detailed guidance on setting up a reverse proxy, let me know.Sources: - GitHub Issue 3961
💡 Hint: Mention RunLLM in the thread for followups.
Chaitanya
·
Also there is a health check I have added for the ECS. It will ping the /healthz endpoint periodically. Actually it should ping /phoenix/v1/healthz but as per below log, its different. INFO: 10.41.248.21:63560 - "GET /phoenix/v1/phoenix/v1/healthz HTTP/1.1" 200 OK
Roger Y.
·
Since the health check is returning a 200 status code, it seems to be functioning as expected and may not be the root cause of the issue. Proxy-related challenges can be complex to diagnose externally, as they often depend on specific vendor configurations. Here’s a link to a similar situation that others have encountered, which might provide some helpful context.
Chaitanya
·
Roger Y. do you want me to try with the debug image here?
Roger Y.
·
are you already using the latest version? in that case you may not need the debug version
Chaitanya
·
Yeah I am using the latest version only. I am using the ALB in AWS. How can I debug the error?
Roger Y.
·
I’m not entirely sure, as I don’t have the infrastructure to replicate the issue. However, could you take a look in your browser and check the URL for modernizr.js as shown in the screenshot below? Would you be able to paste the URL into your browser and experiment with it until it returns the actual file?
Chaitanya
·
I found the file. There is an error "Unexpected token <"
Roger Y.
·
That’s not the actual file, because the html is returned by the load balancer when it’s not found. The actual file should be a text file like below
Chaitanya
·
So I removed the path-patterns in the listener rule configurations and the environment variable for host root path. After that I can see the UI and API are both working. It will be better to have an example for deploying the phoenix in any cloud platform with various settings for paths. Thanks Roger .
🙌1