Efficient Token Breakdown by Model in a Specific Session | Arize AI Community

Arize AI Community Icon

RunLLM
APP
·
The most efficient way in Phoenix is to aggregate token counts at the database level using the backend DataLoaders and cost summary APIs, rather than pulling all spans into a DataFrame and grouping in Python. Phoenix provides async DataLoaders (e.g., SpanCostSummaryBySession, SpanCostSummaryByGenerativeModel) that sum prompt, completion, total, and reasoning tokens by model for a session directly in SQL, which is much faster and scales better than client-side aggregation (span_cost_summary_by_generative_model.py, span_cost_summary_by_project_session.py).However, there is currently no public Python client method that exposes this aggregation directly—you would need to use the GraphQL API or extend the backend to expose a session+model breakdown endpoint. Your current approach is correct for the client, but for large sessions, consider querying the backend for pre-aggregated results or requesting this feature upstream for better performance (span_cost_summary_by_project_session.py). Would you like more detail on how to use the backend DataLoaders or GraphQL for this?Sources: - span_cost_summary_by_generative_model.py - span_cost_summary_by_project_session.py
💡 Hint: Mention RunLLM in the thread for followups.
Ben E.
·
RunLLM Yes, please.
RunLLM
APP
·
Certainly! Here’s a step-by-step breakdown of how Phoenix handles efficient token breakdown aggregation by model for a session, and how you could leverage this: 1. Backend DataLoaders: Phoenix implements async DataLoaders such as SpanCostSummaryByProjectSessionDataLoader and SpanCostSummaryByGenerativeModelDataLoader. These use SQL queries to aggregate token counts (prompt, completion, total, and detailed types like reasoning) grouped by model and session directly in the database, avoiding the need to fetch and process all spans in Python. For example, see the SQL aggregation in span_cost_summary_by_project_session.py and span_cost_summary_by_generative_model.py.2. GraphQL API: The Phoenix backend exposes these aggregations via its GraphQL API. For a session, you can query the costDetailSummaryEntries or costSummary fields on the ProjectSession node, which return pre-aggregated token and cost breakdowns (including prompt, completion, total, and detailed types if available). See the usage in the UI code (SessionTokenCostsDetails.tsx) and the corresponding GraphQL query.3. How to Use: - If you are using the Phoenix UI, these breakdowns are already displayed in the session details and token cost details panels. - If you want to access this programmatically, you can send a GraphQL query to the backend, e.g.:
node(id: $nodeId) { ... on ProjectSession { costDetailSummaryEntries { tokenType isPrompt value { tokens } } } } }
RunLLM
APP
·
- This will return a list of token breakdowns by type for the session, already aggregated.4. Custom Extension: If you need a breakdown by both session and model, and it’s not exposed in the current API, you could extend the backend to add a new resolver that uses the DataLoader pattern to group by both session and model, leveraging the same SQL aggregation logic.5. Why This Is Faster: By aggregating in SQL, you avoid transferring and processing millions of rows in Python, which is the main bottleneck in your current approach.Summary: For best performance, use the backend’s aggregation endpoints (via GraphQL or by extending the backend) rather than client-side DataFrame grouping. If you need a new breakdown (e.g., by both session and model), consider contributing a new resolver to the backend using the existing DataLoader patterns.Sources: - span_cost_summary_by_project_session.py - SessionTokenCostsDetails.tsx
💡 Hint: Mention RunLLM in the thread for followups.
Ben E.
·
RunLLM Let's say I want a breakdown by model, and I'll do the loop over the relevant sessions myself. How can I do that in the simplest way inside a Jupyter notebook?
RunLLM
APP
·
The simplest way in a Jupyter notebook is to use the Phoenix Python client to fetch all spans for each session, then group by model in pandas as you are currently doing. There is no built-in client method to fetch pre-aggregated token breakdowns by model for a session, so you must fetch the spans and aggregate locally (span_cost_summary_by_project_session.py).Your current approach—fetching spans with px.Client().get_spans_dataframe(...) and then grouping by attributes.llm.model_name in pandas—is the recommended method for this use case in a notebook. For multiple sessions, loop over session IDs, fetch spans for each, and aggregate as needed.Sources: - span_cost_summary_by_project_session.py
💡 Hint: Mention RunLLM in the thread for followups.
Mikyo
·
Hey Ben E. good to see you here. Re-constructing sessions via the API is unfortunately a bit slow given it's just doing a linear scan but we do have sessions indexed and we will be adding a session API for sure. But for now with API access I think you have what we have at the moment. I'll file an enhancement when I'm back at my desk tomorrow. With regards to cost, we do have that indexed as well but it's not part of the spans API quite yet. Looks like you want it for a specific session. Def. can add it too. Given you have the token counts you could use something like litellm or openrouter to look up the costs in the notebook: https://docs.litellm.ai/docs/completion/token_usage#5-cost_per_token Sorry to not have a perfect solution for you. Cost and Session support in the REST APIs will definitely be coming (they are in the graphQL APIs)
Ben E.
·
Mikyo Thanks for the quick answer! I just need the token counts, not the actual cost, so we're good on that front 🙂 I'll work with the client for now.
1