Adding Custom Metadata to LLM Spans in Phoenix Tracing

I have an agent with tools that themselves can be agents/query pipelines (I'm using LlamaIndex abstractions)

I'm using Phoenix to get all traces (which is awesome btw!) of my agent's runs

I want to run evals on multiple levels that don't exactly match up with the span, trace, chain, levels of phoenix

so I'm using phoenix spans as a source of truth but then constructing my own eval task in a notebook

The challenge i'm having is being able to disambiguate different spans based on my own qualifiers vs using what phoenix gives out of the box

I want to annotate different LLM spans with some custom attributes I have that I will use downstream in my eval notebook afterwards

What i've tried

passing metadata to my LLMTextCompletionProgram

tried using tracer_api.get_current_span() and then trying to set attributes but this doesn't take the LLM span ! 😢

instead it will update a top level span that I started with with tracer.start_as_current_span("ROOT_AGENT_START") and rest of my agents run is under this span