Hello Arize team, I have two questions: 1 - Did something happen / change with the query explode? When I run this command, I get an output:
query = SpanQuery().where(
"span_kind" == "RERANKER",
).select(
# input="reranker.query",
model = "reranker.model_name",
)
reranked_docs_df = px.active_session().query_spans(query)
reranked_docs_dfHowever, if I run this command, I don't get anything back:
query = SpanQuery().where(
"span_kind" == "RERANKER",
).select(
# input="reranker.query",
model = "reranker.model_name",
).explode(
"reranker.output_documents",
reference = "reranker.document_content",
)
reranked_docs_df = px.active_session().query_spans(query)
reranked_docs_dfIf I switch the reference in the select, I can get the result of the list of documents for the output contents:
query = SpanQuery().where(
"span_kind" == "RERANKER",
).select(
# input="reranker.query",
model = "reranker.model_name",
reference = "reranker.output_documents"
)
reranked_docs_df = px.active_session().query_spans(query)
reranked_docs_df2 - What is the suggested way of performing evaluations when query transformation is in place (same query asked multiple times in different ways) like in the picture attached? I do want to calculate the DCG@5, Precision@5 and Hit rate for the retrieval part, however I am trying to think of the best way to capture the retrieval evaluation and be considerate of the cost considering I'm getting 100s of documents per question.
concat for retrieval not working as well
Hi Teodor, I'm not super familiar with our query DSL, but as far as I understand we haven't touched our query logic for a few months. Were these queries working before as intended?
yes, i used them a couple of months ago successfully
https://arize-ai.slack.com/archives/C04R3GXC8HK/p1707131620601149 i had a discussion on this channel about it as well
ah ok, February 5th was just before we switched over from our older instrumentation code to OpenTelemetry-compliant OpenInference instrumentation, it's possible that after upgrading your span topology has changed because of the way OpenTelemetry infers the parent span
I'd also double check to make sure the semantic conventions haven't changed, you can find them here: https://github.com/Arize-ai/openinference/blob/main/python/openinference-semantic-conventions/src/openinference/semconv/trace/__init__.py
seems to be the same:
class RerankerAttributes:
"""
Attributes for a reranker
"""
RERANKER_INPUT_DOCUMENTS = "reranker.input_documents"
"""
List of documents as input to the reranker
"""
RERANKER_OUTPUT_DOCUMENTS = "reranker.output_documents"
"""
List of documents as output from the reranker
"""
RERANKER_QUERY = "reranker.query"
"""
Query string for the reranker
"""
RERANKER_MODEL_NAME = "reranker.model_name"
"""
Model name of the reranker
"""
RERANKER_TOP_K = "reranker.top_k"
"""
Top K parameter of the reranker
"""can you try this one with document.content instead of reranker.document_content?
query = SpanQuery().where(
"span_kind" == "RERANKER",
).select(
# input="reranker.query",
model = "reranker.model_name",
).explode(
"reranker.output_documents",
reference = "document.content",
)
reranked_docs_df = px.active_session().query_spans(query)
reranked_docs_dfwhat do you get if you run this?
query = SpanQuery().where(
"span_kind" == "RERANKER",
).select(
reference = "reranker.output_documents",
)
reranked_docs_df = px.Client().query_spans(query)
print(type(reranked_docs_df.reference[0][0]))
print(reranked_docs_df.reference[0][0].keys())ok, so it does have document.content
does it work without model name?
query = SpanQuery().where(
"span_kind" == "RERANKER",
).explode(
"reranker.output_documents",
reference = "document.content",
)
reranked_docs_df = px.active_session().query_spans(query)
reranked_docs_dfnope, it seems that the moment i add .explode() or .concat() it completely stops returning anything
