Getting Span ID for Feedback in Django with LangChain Instrumentation

RunLLM this example here seems relevant except I’m not using LangGraph and not sure how to adapt it to my use of LangChain’s RunnableWithMessageHistory

Hey Andrew S. can you maybe highlight where you are calling get_current_span ? I might not have the time to repro your issue today but I'm gonna cc John G. - we're working on better documenting this pattern of sending feedback at inference time.

sure, the workflow is like this:

 with get_openai_callback() as callback:
            async for chunk in rag_chain.astream

I tried getting the span in the with get_openai_callback() as callback context and also the async for chunk loop. I basically tried getting the span everywhere, including he function that creates the chain. If there is an example that uses langchain instead of langgraph, that might be helpful.

we create the RunnableWithMessageHistory like so:

    RunnableWithMessageHistory().with_config(
        metadata=metadata,
        run_name=run_name,
        run_id=uuid.UUID(str(chat_run_id)),
    )

we assign a run id there but it’s not the same trace ID that is shown by Phoenix.

Hey Andrew S. - agree this is a bit confusing and we need a bit more examples. The engineer that added this utility is out so I'll try to see if anyone on the team can try to help. If it helps at all this is the commit that added this functionality (https://github.com/Arize-ai/openinference/commit/b4d78b0c2e48558c4e55ba3345badef22034f693) Sorry gotta run to a meeting but will follow-up if I can find an example.

Thanks, 🔒[private user]! I tried your solution but still having issues. Just to clarify, should the trace ID in Phoenix correspond to the run_id assigned here:

    RunnableWithMessageHistory().with_config(
        metadata=metadata,
        run_name=run_name,
        run_id=uuid.UUID(str(chat_run_id)),
    )

They don’t match up and I’m wondering if Phoenix is somehow not respecting the custom run_id that we assign to our chain. gonna try this to next to see if I can get a run ID that matches up with the trace ID: https://docs.smith.langchain.com/observability/how_to_guides/trace_with_langchain#access-run-span-id-for-langchain-invocations

Maybe the run id we add to the chain config is being used for the session id in Phoenix as opposed to the trace id. I’ll check that

Anyways, all the more reason to refactor and migrate away from langchain.

figured it out. adding this callback when I invoke the chain did the trick:

class LogSpansCallback(BaseCallbackHandler):
    def on_llm_end(
        self,
        response: LLMResult,
        *,
        run_id: UUID,
        parent_run_id: Optional[UUID] = None,
        **kwargs: Any,
    ) -> Any:
        log_spans(run_id, "on_llm_end")

    def on_llm_start(
        self,
        serialized: dict[str, Any],
        prompts: list[str],
        *,
        run_id: UUID,
        parent_run_id: Optional[UUID] = None,
        tags: Optional[list[str]] = None,
        metadata: Optional[dict[str, Any]] = None,
        **kwargs: Any,
    ) -> Any:
        log_spans(run_id, "on_llm_start")

        with get_openai_callback() as callback:
            async for chunk in rag_chain.astream(
                data,
                {
                    "configurable": {"session_id": chat_session_id},
                    "callbacks": [LogSpansCallback()],
                },
            ):

Hey Andrew S. - really appreciate you digging in. I'm not an expert of LangChain so I'm glad you figured it out. In general you are correct - the run_ids in langchain are not the span or trace_ids since LangChain doesn't use opentelemetry. We maintain a mapping that you can resolve at runtime. Please let Roger Y. know if there are any ergonomic improvements we can make here and John G. and I will ensure we document this better. Thanks a bunch!