Visualizing User Embeddings in Phoenix for RAG Pipelines

·Nov 04, 2024 04:56 PM

I think this is an elementary question but I could not find a definitive answer in the documentation. I have implemented (manually instrumented) tracing for my application. I can see all my traces and spans come in nicely as I expect. Suppose some of the traces coming in are for a standard RAG pipeline. I would like to visualize all of the embeddings for the users' questions using Phoenix's awesome UMAP visualization. After spending a couple days with the documentation, it seems to me like the best path forward is to query the spans to a dataframe and then import the dataframe as a dataset with a schema. Does that sound right? I guess I was holding out that there was a way to grab all embeddings with a specific span name from the project view. Thanks in advance for you engagement!

4 comments

· Sorted by Oldest

RunLLM
APP
·
Yes, querying spans to a DataFrame and then importing it as a dataset with a schema is a valid approach. You can use the get_spans_dataframe method to extract spans and their embeddings into a DataFrame. Then, define a schema to specify which columns represent embeddings and import this DataFrame into Phoenix for visualization. If you want to filter spans by a specific span name, you can use the query DSL to select spans with the desired attributes. For example, you can filter for spans with a specific span_kind and extract the embeddings. Would you like more detail on how to implement this?Sources: - Extract Data from Spans - Export Your Data
💡 Hint: Mention RunLLM in the thread for followups.
Roger Y.
·
best path forward is to query the spans to a dataframe and then import the dataframe as a dataset with a schema. Does that sound right?
yea, that’s the only way to do that right now
Matthew B.
·
Thanks Roger Y. for confirming. That might be a nice-to-have feature. One could quickly visually spot-check all the embeddings with a specific span name, e.g. "user-input" or "retrieved-documents", and make sure there aren't outliers or collapsing.
Roger Y.
·
np. would you mind creating an enhancement request for us to track?
👍🏽1