Tracking Non-LLM Costs for Cumulative Span Analysis
Hi everyone, is there a way to track non llm costs and add them to the cumulative cost of the span ? If so, what is the attribute I need to set with the cost ?
Hi everyone, is there a way to track non llm costs and add them to the cumulative cost of the span ? If so, what is the attribute I need to set with the cost ?
It seems likely that Phoenix currently tracks LLM costs automatically using token counts and model pricing, but there is no documented, built-in attribute for reporting arbitrary non-LLM costs directly on a span. Enhancement requests exist for supporting custom cost reporting on spans for external services, but this is not yet a standard feature or convention in the released platform or semantic conventions documentation. As of now, only LLM-related costs (using token counts and model info) are supported for automatic aggregation and display, and there is no official attribute for manual non-LLM cost injection into the cumulative cost calculation for a span or trace. If you want to track non-LLM costs, you may need to use custom span attributes (e.g., a user-defined attribute like custom.cost) for your own analysis, but these will not be included in Phoenix's built-in cost aggregation or UI displays. For more details or updates on this feature, see the related enhancement discussion and consider following or contributing to the relevant GitHub issue for custom cost support. Would you like a step-by-step breakdown or code example for adding custom attributes to spans?Sources: - Cost tracking for function calls and external services (enhancement request)
馃挕聽Hint: Mention RunLLM in the thread for followups.
馃敀[private user] yes, I've looked at the issue and it is exactly what I want, I'll follow it. Thank you for the answer !
Mikyo I don't think I'll open a new issue since there is already a similar one. I want to track the costs of my agentic application (with adk), most of the costs are LLM costs which are already handled, but sometimes, the tools do stuff that incurs additional costs (image generation for example), and I want to track them in the cumulative cost of the trace. Currently I just add a new attribute cost in the span of the tool, but it isn't taken into account in the total sum.
Mikyo yeah, keeping them clearly separated (tool costs and llm costs for example as you said) is what I want, as well as having them aggregated in the cumulative cost. Is it possible to use the span cost table currently ? Here's an example of how I do it for now :
span = tracer.start_span("image_generation")
span.set_attribute("cost", 0.245)
span.end()
Gotcha, no it鈥檚 not currently supported so would definitely need to make adjustments. I鈥檒l update the ticket as I go. Appreciate the enhancement idea!
