Issues with Annotations on Hosted Phoenix Endpoint Calls

Jarrad M. · 2024-10-04T10:46:44.047Z

Hi team! I'm using hosted Phoenix, and implemented annotations 3-4 weeks ago. Everything was working fine. Checked in today and it seems my calls to the /v1/span_annotations endpoint are now returning a 200 response but also the HTML doc of a login page. ✅ tests fired from https://app.phoenix.arize.com/apis work: 200 response and the annotation goes through ❌ fetch from my own node.js server: 200 response and no annotation goes through, get a HTML doc back Any idea what has changed that might have caused this?

RunLLM
APP
·
The issue you're experiencing is likely due to authentication changes in the hosted Phoenix platform. According to the authentication setup documentation, Phoenix requires proper authentication headers to be set when making API calls. If these headers are missing or incorrect, the server may redirect to a login page, resulting in the HTML document you're seeing.To resolve this, ensure that your Node.js server is setting the correct authentication headers. Specifically, you need to set the Authorization header with the format Bearer <token> . Here is an example of how to set this header in a Node.js fetch request: LBKSconst fetch = require('node-fetch');LBKSconst response = await fetch('[https://app.phoenix.arize.com/v1/span_annotations](https://app.phoenix.arize.com/v1/span_annotations)', { method: 'POST', headers: { 'Content-Type': 'application/json', 'Authorization': `Bearer ${process.env.PHOENIX_API_KEY}` }, body: JSON.stringify({ data: [ { span_id: "your_span_id", name: "your_annotation_name", annotator_kind: "HUMAN", result: { label: "your_label", score: your_score, explanation: "your_explanation" }, metadata: {} } ] }) });LBKSconst result = await response.json(); console.log(result);LBKSWould you like more detail on setting up authentication or troubleshooting this issue further?Sources: - Authentication Setup
💡 Hint: Mention RunLLM in the thread for followups.
Dustin N.
·
Hi Jarrad M.! I’m sorry you’re running into issues, I’ll forward this to the team that manages hosted Phoenix so they can look into it
Mikyo
·
Any luck with setting the headers Jarrad M. ?
Kiko C.
·
Hi Jarrad M. , sorry for the issues you’re facing. We are working on bug fixes as we speak so you experience should improve in the coming days
Jarrad M.
·
Hey 🔒[private user], appreciate you looking into this. So I'm clear in my understanding: there's a known bug that means the REST api is returning a 404 on known span IDs, but another bug means that that 404 is not returned to me and I instead am getting a 200 response. Is that a correct summary of the current state?

Jarrad M.

totally, I was able to reproduce it in postman with this:


curl --location 'https://app.phoenix.arize.com/v1/span_annotations?sync=false' \
--header 'Authorization: Bearer api_key_here' \
--header 'api_key: api_key_here' \
--header 'PHOENIX_CLIENT_HEADERS: api_key=api_key_here' \
--header 'Content-Type: application/json' \
--data '{
  "data": [
    {
      "span_id": "a2f0cd65be677102",
      "name": "string",
      "annotator_kind": "HUMAN",
      "result": {
        "label": "test",
        "score": 0,
        "explanation": "test"
      },
      "metadata": {}
    }
  ]
}'

Pretty sure I have tried every permutation of headers, same thing every time. Hopefully I am making a basic mistake somewhere!

Jarrad M.
·
no sweat, appreciate what you guys are doing
1
Jarrad M.
·
Hey team, I've got my production code attempting (and still failing) to programmatically log annotations. It's falling back to storing the annotation elsewhere for me to manually insert with the UI. Anyone have any further clarity on whether there is a bug here, or if I am doing something wrong with those headers? This is re hosted phoenix, and was working up until 2 maybe 3 weeks ago.
Jarrad M.
·
oh great! wasn't sure if the mistake was still on my end. Will keep an eye on things, good luck.

Kiko C.

This is the request using curl from our API page:

curl -X 'POST' \
  'https://llamatrace.com/v1/span_annotations?sync=false' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "data": [
    {
      "span_id": "***",
      "name": "test_annotation",
      "annotator_kind": "HUMAN",
      "result": {
        "label": "is_test",
        "score": 1,
        "explanation": "string"
      },
      "metadata": {}
    }
  ]
}'

Granted you don't see the authentication headers since I'm running the request from the API UI. However this did not work when you raised your concerns. Gonna try from terminal now.

Kiko C.
·
Hey Jarrad M. was able to reproduce the authentication error and have a PR out. I expect to unblock you by tomorrow
Jarrad M.
·
Heck yes, legends. Thank's heaps!
Kiko C.
·
Fix is live, let me know if you encounter issues

Jarrad M.

Kiko C. Still no luck unfortunately. Pretty sure I've tried every which way, here's an example of what I'm testing in postman:

curl --location 'https://app.phoenix.arize.com/v1/span_annotations' \
--header 'accept: application/json' \
--header 'Content-Type: application/json' \
--header 'OTEL_EXPORTER_OTLP_HEADERS: api_key=redacted' \
--header 'PHOENIX_CLIENT_HEADERS: api_key=redacted' \
--header 'PHOENIX_COLLECTOR_ENDPOINT: https://app.phoenix.arize.com' \
--header 'Authorization: Bearer redacted' \
--header 'api_key: redacted' \
--data '{
  "data": [
    {
      "span_id": "a2f0cd65be677102",
      "name": "string",
      "annotator_kind": "HUMAN",
      "result": {
        "label": "test",
        "score": 0,
        "explanation": "test"
      },
      "metadata": {}
    }
  ]
}'

For context, initial spans are being traced from my DSPy-instrumented backend (this is working), I then pass the span ID to a nextjs app frontend where the user gives feedback. The annotation is submitted from there, and this is where I'm running into the error. This was all working a few weeks ago, so I wonder if some of the recent auth changes on the phoenix end have killed my API key or something? Is there a way I can generate a fresh one?

Jarrad M.
·
Ahh no dice. replaced with a span ID of mine, and my api key, and I'm still getting a 200 response but with the html login screen as the body, and no annotation goes through

23 comments