Hi team - I've added manual feedback labels in the Phoenix UI and can see them there. How can I retrieve these specific labels programmatically using the Python client? Tried `query_spans` and `get_evaluations` based on docs/GH issues, but they don't seem to return the UI-added labels. Is there a specific function or syntax I should use? Thanks!
It seems likely that to retrieve feedback labels added in the Phoenix UI programmatically, you should use the get_spans_dataframe method from the Phoenix Python client. This method allows you to download all spans as a DataFrame, which should include the annotations and feedback labels you added in the UI. Here's a basic example of how you might use this method: LBKSimport phoenix as pxLBKSclient = px.Client() spans_df = client.get_spans_dataframe(project_name='your_project_name') print(spans_df)LBKSThis will give you a DataFrame containing all the spans, including any annotations or feedback labels. If you need more detailed guidance, let me know!Sources:
- https://github.com/Arize-ai/phoenix/issues/5652
💡 Hint: Mention RunLLM in the thread for followups.
We tried running client.get_spans_dataframe(project_name='transcript-agent-mvp') as you suggested. However, when we inspected the columns of the resulting DataFrame (spans_df.columns), we didn't see any columns that seem to represent the manual feedback labels/annotations we added through the UI (like 'Pass/Fail', 'Correctness', etc.). The columns primarily contained standard span attributes and LLM details.
i can see these in the UI
Hey Hugo, I empathize with your confusion that pulling spans doesn't contain the annotations. In our data model spans and the correlated annotations are separate entities - the span being a runtime artifact that is immutable and the spans being "annotations" that are the results of judgment by llms, humans, and code. For that reason anything called get_spans in the API is actually just that - it gets the spans. However we understand that getting the annotations at the same time is probably 90% of the case so we will be making ergonomic changes to the client and API. We have a major overhaul on annotations coming in the following week or so an a lot of this will be made more clear. https://github.com/Arize-ai/phoenix/issues/5917 Somewhat confusingly I think this trace dataset method would contain both (https://docs.arize.com/phoenix/tracing/how-to-tracing/importing-and-exporting-traces/extract-data-from-spans#save-all-traces) I'll work with the team to get this sorted out this week.
thanks for the prompt reply! Following your suggestion, I tried using client.get_trace_dataset(project_name='transcript-agent-mvp') again and inspected the returned `trace_dataset` object. Here's what I found: 1. The trace_dataset.dataframe was populated (125 spans, 32 columns), but inspecting the columns didn't reveal any obvious ones for the manual annotations added in the UI (like 'Final Answer Quality', 'SQL Correctness', etc.). 2. The trace_dataset.evaluations list was still empty. This seems to confirm that the UI annotations aren't directly exposed via these standard retrieval methods currently. While I understand the annotation overhaul (#5917) is coming and will improve this, is there any current method or workaround available today (even if less ergonomic) that would allow us to programmatically retrieve these specific UI-added annotations? We'd like to use them for analysis now if possible. Thanks for your help!
Sorry have been AFK and will be for the rest of the day. There might have been some oversight in our API footprint where the annotations are not fetch-able right now. It's a bit surprising to me. You can definitely pull the ones that are sent in programatically. If there's a gap here my apologies. We'll get it fixed in the next day or two.
if the number of spans is not very large, you can use the following script as a workaround for the time being. I attached a screenshot to show that the output is a dataframe
import pandas as pd
from httpx import Client
base_url = "http://localhost:6006"
# Replace with your actual project ID
project_id = "UHJvamVjdDox"
query = """
query ($id: GlobalID!, $first: Int, $after: String) {
node(id: $id) {
... on Project {
spans(first: $first, after: $after) {
edges {
node {
spanId
spanAnnotations {
name
label
score
explanation
annotatorKind
metadata
}
}
cursor
}
pageInfo {
hasNextPage
endCursor
}
}
}
}
}
"""
def fetch_all_spans(client, project_id):
all_spans = []
has_next_page = True
cursor = None
while has_next_page:
# Make the GraphQL request
resp = client.post(
"/graphql",
json={
"query": query,
"variables": {
"id": project_id,
"first": 1000, # Fetch 1000 spans at a time
"after": cursor,
},
},
)
data = resp.json()["data"]["node"]["spans"]
edges = data["edges"]
page_info = data["pageInfo"]
# Extract spans with annotations
spans = [x["node"] for x in edges if x["node"]["spanAnnotations"]]
all_spans.extend(spans)
# Update pagination info
has_next_page = page_info["hasNextPage"]
cursor = page_info["endCursor"]
return all_spans
# Create client and fetch all spans
client = Client(base_url=base_url)
all_spans = fetch_all_spans(client, project_id)
# Normalize the data into a DataFrame
spans_df = pd.json_normalize(all_spans)
# Explode the annotations
exploded = spans_df.explode("spanAnnotations")
# Normalize the annotations
annotations = pd.json_normalize(exploded["spanAnnotations"])
# Set the span ID as the index
annotations.index = exploded["spanId"]
annotations
