Is openinference-instrumentation-litellm expected to log all tools available to LLM in the trace? I'm seeing an example where the tools aren't logged while the LLM output is a tool call, so I wonder what tools are available to the LLM. With openinference-instrumentation-openai, there is usually a Tools tab besides Input Messages , I wonder if this should also be the case for litellm.
No, the current openinference-instrumentation-litellm does not log all tools available to the LLM in the trace; it only logs tool calls when they appear in the model's output, not the set of available tools at invocation. This is a known limitation and differs from openinference-instrumentation-openai, which does log available tools and enables the "Tools" tab in Phoenix [source]聽[source].To display available tools in the Phoenix UI for LiteLLM traces, the instrumentation would need to be extended to log the set of tools provided to the LLM at the start of the call, similar to the OpenAI instrumentation. This is not currently implemented [source]聽[source].Would you like more detail on how this could be implemented or tracked?Sources:
- https://github.com/Arize-ai/openinference/issues/1800
- https://github.com/Arize-ai/openinference/issues/1750
- https://github.com/Arize-ai/openinference/issues/1751
馃挕聽Hint: Mention RunLLM in the thread for followups.
Thank you, yeah, I find this feature of showing all available tools to an agent super helpful, esp. for an agent that I'm not familiar with. More context: I was trying to understand https://github.com/robusta-dev/holmesgpt better, which used litellm, then I forked it https://github.com/zyxue/holmesgpt/commit/d0fc63738eabf870b08d66dd37bcb75040ab1ef2 and added phoenix tracing, it's great that I can see what the system prompt is sent to LLM now, but also realizing not seeing what tools are available.
