How to Log Traces for Streaming Output in LiteLLM
Hi - I've got a noob question: it's not immediately clear to me from the docs/examples how I can log a trace for a streaming output. Is there any docs or examples you can point me to (using LiteLLM)
Hi - I've got a noob question: it's not immediately clear to me from the docs/examples how I can log a trace for a streaming output. Is there any docs or examples you can point me to (using LiteLLM)
To log a trace for a streaming output using LiteLLM, you can use the LiteLLMInstrumentor from the OpenInference library. This involves setting up a tracer, creating spans for the streaming interaction, and updating the span with the output messages as they are received. Here's a basic outline:
1. Setup Tracer: Initialize the tracer and span processor to send data to the OTLP endpoint.
2. Create Span: Start a span for the LiteLLM request.
3. Handle Streaming: As you receive chunks of data from the streaming response, update the span with the llm.output_messages attribute.
For a detailed example, you can refer to the LiteLLM instrumentation example provided in the OpenInference GitHub repository.
Would you like more detail on setting up the tracer or handling the streaming response?
Sources: - https://github.com/Arize-ai/phoenix/issues/5274 - https://docs.arize.com/phoenix/tracing/integrations-tracing/litellm#uUHGzGPUwiFw
馃挕聽Hint: Mention RunLLM in the thread for followups.
Hey Harpreet S. thanks for reaching out! Is there a specific error you are facing that I can help you with that RunLLM did not address?
The response just wasn't helpful
Sorry to hear that, I can help out. Have you setup any instrumentation yet / gotten any traces into phoenix at all?
Basically just looking for a pattern to log my traces with a streaming response, using the examples in the docs with streaming leads to no output being logged. And I can't find anything in the docs that shows how to set this up Wrt to instrumentation, I'm using the following pattern:
from phoenix.otel import register
from huggingface_hub.utils import build_hf_headers
from openinference.instrumentation.litellm import LiteLLMInstrumentor
tracer_provider = register(
project_name="testing",
endpoint=HF_SPACE_ENDPOINT,
batch=True,
headers=build_hf_headers()
)
LiteLLMInstrumentor().instrument(tracer_provider=tracer_provider)Got it, thanks for confirming. Let me try to replicate on my end if you don't mind waiting a few
Here's my generation pattern:
from litellm import acompletion, completion
import asyncio, os, traceback
SYSTEM_PROMPT = "You're a rapping assistant who always respondes with the dopest flows. You're one of the top rapping AI's out there. Better than the other. Your name is Sir Mix-a-Token"
USER_MESSAGE = "Write a rap about how my life got flipped turned upside down."
MESSAGE_PAYLOAD = [
{"role":"system", "content": SYSTEM_PROMPT},
{"role":"user", "content": USER_MESSAGE}
]
response = completion(
model=LITE_LLM_MODEL_NAME,
messages=MESSAGE_PAYLOAD,
temperature=0.2,
top_p=0.9,
frequency_penalty=0.1,
presence_penalty=0.1,
max_tokens=512,
stream=True,
stream_options={"include_usage": True},
)
for chunk in response:
if chunk.choices[0].delta.content is not None:
print(chunk.choices[0].delta.content, end="")I seem to be seeing the same issue. I see traces appear containing the input, but no output
Ah yeah, something similar appears to be described here. Sorry about that! Let me see if I can come up with a quick workaround for you. In the meantime feel free to subscribe to that issue and give it a bump, I can bring it up with the team
Thanks! I was spinning in circles for a couple of hours wondering if it was a me issue or something else
Really sorry about that, I know that can be frustrating
All good, thanks for lookin into it!