Hi Mikyo John G. I dig in more, and realize it has more to do with the anthropic llm auto-instrumentation - sometimes the llm span just isn't created, and it seems to be related to how fast the requests are being sent - whether it's triggered by human one after another slowly - all llm span can be captured, or programmatically one after another (which is apparently faster than human) - llm span can be dropped. We are using Anthropic streaming, and that's not supported in Phoenix at the moment, so we do the following:
stream = await self._anthropic_client.messages.create(
max_tokens=self._max_tokens,
system=self._system_prompt,
model=self._model,
messages=history.messages,
thinking=thinking,
tools=history.client_tools,
stream=True,
extra_headers=get_aigateway_extra_headers({"conversation-id": str(history.id)}),
)Our colleague Deepankar M. has created a feature request for anthropic stream method to be supported - https://github.com/Arize-ai/openinference/issues/1343 Would it be possible to fasten up this request to see if it solves the above issue? Thanks! 🙏
