Hi, I'm encountering some slowness on my local Phoenix.
💡 Hint: Mention RunLLM in the thread for followups.
This is the query that's taking a while (~3 minutes in this case):
phoenix-1 | 2025-06-07 16:01:36,465 INFO sqlalchemy.engine.Engine BEGIN (implicit)
phoenix-1 | 2025-06-07 16:01:36,466 INFO sqlalchemy.engine.Engine SELECT traces.project_rowid, coalesce(sum(spans.llm_token_count_prompt), ?) AS prompt, coalesce(sum(spans.llm_token_count_completion), ?) AS completion, coalesce(sum(spans.llm_token_count_prompt), ?) + coalesce(sum(spans.llm_token_count_completion), ?) AS total
phoenix-1 | FROM traces JOIN spans ON traces.id = spans.trace_rowid
phoenix-1 | WHERE spans.start_time >= ? AND traces.project_rowid IN (?, ?, ?) GROUP BY traces.project_rowid
phoenix-1 | 2025-06-07 16:01:36,466 INFO sqlalchemy.engine.Engine [generated in 0.00012s] (0, 0, 0, 0, '2025-06-07 15:46:00.000000', 3, 3, 3)
phoenix-1 | 2025-06-07 16:04:25,489 DEBUG sqlalchemy.engine.Engine Col ('project_rowid', 'prompt', 'completion', 'total')
phoenix-1 | 2025-06-07 16:04:25,492 DEBUG sqlalchemy.engine.Engine Row (3, 918256, 18650, 936906)
phoenix-1 | 2025-06-07 16:04:25,493 INFO sqlalchemy.engine.Engine COMMITThis is the query itself:
SELECT traces.project_rowid, coalesce(sum(spans.llm_token_count_prompt), ?) AS prompt, coalesce(sum(spans.llm_token_count_completion), ?) AS completion, coalesce(sum(spans.llm_token_count_prompt), ?) + coalesce(sum(spans.llm_token_count_completion), ?) AS total
FROM traces JOIN spans ON traces.id = spans.trace_rowid
WHERE spans.start_time >= ? AND traces.project_rowid IN (?, ?, ?) GROUP BY traces.project_rowidAnd the query plan:
sqlite> .eqp on
sqlite> SELECT traces.project_rowid, coalesce(sum(spans.llm_token_count_prompt), 0) AS prompt, coalesce(sum(spans.llm_token_count_completion), 0) AS completion, coalesce(sum(spans.llm_token_count_prompt), 0) + coalesce(sum(spans.llm_token_count_completion), 0) AS total FROM traces JOIN spans ON traces.id = spans.trace_rowid WHERE spans.start_time >= '2025-06-07 15:46:00.000000' AND traces.project_rowid IN (3) GROUP BY traces.project_rowid;
QUERY PLAN
|--SEARCH traces USING COVERING INDEX ix_traces_project_rowid (project_rowid=?)
`--SEARCH spans USING INDEX ix_spans_trace_rowid (trace_rowid=?)
3|918256|18650|936906I made this change locally, and Phoenix is super-snappy again:
diff --git a/src/phoenix/server/api/types/Project.py b/src/phoenix/server/api/types/Project.py
index fe3876a95..bed722283 100644
--- a/src/phoenix/server/api/types/Project.py
+++ b/src/phoenix/server/api/types/Project.py
@@ -149,6 +149,7 @@ class Project(Node):
time_range: Optional[TimeRange] = UNSET,
filter_condition: Optional[str] = UNSET,
) -> float:
+ return 0
return await info.context.data_loaders.token_counts.load(
("total", self.project_rowid, time_range, filter_condition),
)
@@ -160,6 +161,7 @@ class Project(Node):
time_range: Optional[TimeRange] = UNSET,
filter_condition: Optional[str] = UNSET,
) -> float:
+ return 0
return await info.context.data_loaders.token_counts.load(
("prompt", self.project_rowid, time_range, filter_condition),
)
@@ -171,6 +173,7 @@ class Project(Node):
time_range: Optional[TimeRange] = UNSET,
filter_condition: Optional[str] = UNSET,
) -> float:
+ return 0
return await info.context.data_loaders.token_counts.load(
("completion", self.project_rowid, time_range, filter_condition),
)(obviously not proposing this patch / only useful as a diagnostic)
Definitely let me know if there are other diagnostics that would be helpful.
If you tell me the right answer to this question, I'll make sure I don't make the same mistake twice!
Feedback
Type the right answer here...
↵ Please 'enter' to submit
10.9.1
Got it, thank you
The is the relevant query on the dashboard:
{"query":"query ProjectsPageProjectMetricsQuery(\n $id: ID!\n $timeRange: TimeRange!\n) {\n project: node(id: $id) {\n __typename\n ... on Project {\n traceCount(timeRange: $timeRange)\n latencyMsP50: latencyMsQuantile(probability: 0.5, timeRange: $timeRange)\n tokenCountTotal(timeRange: $timeRange)\n }\n id\n }\n}\n","variables":{"id":"UHJvamVjdDoz","timeRange":{"start":"2025-06-08T18:38:00.000Z"}}}and this is the one on the project page itself:
{"query":"query ProjectPageQuery(\n $id: ID!\n $timeRange: TimeRange!\n) {\n project: node(id: $id) {\n __typename\n ...ProjectPageHeader_stats\n ...StreamToggle_data\n id\n }\n}\n\nfragment ProjectPageHeader_stats on Project {\n traceCount(timeRange: $timeRange)\n tokenCountTotal(timeRange: $timeRange)\n tokenCountPrompt(timeRange: $timeRange)\n tokenCountCompletion(timeRange: $timeRange)\n latencyMsP50: latencyMsQuantile(probability: 0.5, timeRange: $timeRange)\n latencyMsP99: latencyMsQuantile(probability: 0.99, timeRange: $timeRange)\n spanAnnotationNames\n documentEvaluationNames\n id\n}\n\nfragment StreamToggle_data on Project {\n streamingLastUpdatedAt\n id\n}\n","variables":{"id":"UHJvamVjdDoz","timeRange":{"start":"2025-06-08T18:50:00.000Z"}}}This is where the database query is constructed in the server: https://github.com/Arize-ai/phoenix/blob/ccef63a6fd2f6f16b5e054f7063c4f4a5ff6a6f4/src/phoenix/server/api/dataloaders/token_counts.py#L97-L124
