We're getting this error a lot today. Is this something you've seen before?
Could you provide more details about the error message you're encountering? This will help in identifying the issue and providing a more accurate solution. If possible, include any error codes or specific messages that appear.Sources: - https://github.com/Arize-ai/phoenix/issues/6964 - https://arize-ai.slack.com/archives/C04R3GXC8HK/p1718648900.283399
馃挕聽Hint: Mention RunLLM in the thread for followups.
Unexpected token 'u', "upstream c"... is not valid JSONDo you see anything in your browser's network tab or console when this error occurs? What actions were you performing in the UI when the error is triggered?
Curious if it is a network or proxying issue based on what looks to be a plain/text response to a request that the UI expects to return JSON
don't see anything obvious. My guess is that it's because this project has a lot of data in it, if we go to other projects in the same instance, they seem to be fine
that is a good insight. Would you be comfortable sharing how many traces and tokens the project has?
1864 traces, 43M tokens
also wondering if this relates to concurrency, at least two of us were trying to run experiments (with parallelism) at the same time
do you know if your Phoenix instance is using Postgres?
I believe it is? do you mean this might be related to whether it handles json natively in the backend db-?
Not necessarily, just narrowing down what technologies are in play as it impacts whether this may be a Phoenix performance problem
Sqlite is the default if postgres is not configured, and that comes with different performance characteristics than postgres
sure, sqlite is what we use when we run it locally, but this is a hosted shared instance.
I updated the ticket here with the full traceback we were able to pull from the logs: https://github.com/Arize-ai/phoenix/issues/7193
Excellent, thank you! That will be helpful
