Creating Custom query_df and Handling Sub-Query Results in LlamaIndex

·Sep 19, 2023 09:47 AM

Hi Arize, I hope you are all doing well. Im going through this tutorial https://github.com/Arize-ai/phoenix/blob/main/tutorials/llama_index_search_and_retrieval_tutorial.ipynb I have 2 questions:

1.
For the query_df , is there a way to generate your own query_df from scratch? I am trying to retorfit my "sample_query_df" and replace it with the "query_df". At the moment you just download the parquet file from your google cloud.
2.
I am using LlamaIndex's SubQuestionQueryEngine which effectively breaks down the prompt/question into multiple sub-queries with each sub-query having its own query_engine and hence it returns multiple documents rather than standard 2. Have you thought of what the best way you can handle this use case? Thanks in advance.

10 comments

· Sorted by Oldest

Mikyo
·
For the query_df , is there a way to generate your own query_df from scratch? I am trying to retorfit my "sample_query_df" and replace it with the "query_df". At the moment you just download the parquet file from your google cloud.
I may be mis-understanding your question but of-course! You just need to output the inferences from the callback to file(s) after the queries are run. Are you running your engine in production?
Mikyo
·
1.
I am using LlamaIndex's SubQuestionQueryEngine which effectively breaks down the prompt/question into multiple sub-queries with each sub-query having its own query_engine and hence it returns multiple documents rather than standard 2. Have you thought of what the best way you can handle this use case? Thanks in advance.
I think it depends on how the sub-queries are executed - what are you executing the queries against? Are they the same index / vector store? If so you should be able to accumulate the documents but I can't with 100% confidence say we've tested this flow. Let us know if you hit road blocks.
Mikyo
·
More generically, yes, we are thinking about supporting more complex query engines. We are working on application traces which should show you the traces for each sub-query. https://docs.arize.com/phoenix/concepts/langchain-and-llamaindex-traces
Steve H.
·
Mikyo thanks for your response. Regarding the "sample_query_df" here is a snippet of the code supplied : query_df = pd.read_parquet( "http://storage.googleapis.com/arize-assets/phoenix/datasets/unstructured/llm/llama-index/arize-docs/query_data_complete3.parquet", ) query_ds = px.Dataset.from_open_inference(query_df) session = px.launch_app(primary=query_ds, corpus=database_ds) The problem is that query_df is based on something that you have already pre-populated and it has a certain structure. When I try to modify the code base so that rather than use query_df I retrofit sample_query_df (which created from me running the queries myself) I get issues and errors.
Steve H.
·
Mikyo regarding the second issue of complex query engines, in my use case I have multiple indices with each index having its own dedicated tool/query_engine and then having an over arching subqueryengine. I think what I am observing is is returning more than 2 documents, ie) for a given question it will break that down into say 4 sub-queries and each of these sub-queries will retrieve 2 documents, then when we use the classify_relevance function I am not sure it would work properly because I assume it is comparing the original parent query against the documents returned from the sub-queries. Hopefully that makes sense, but to be perfectly honest this seems to be a complex use case. I am just thinking out loud here...but maybe you will need to have some way of getting the relevance of all the sub queries and their correponding retrieved documents. Also maybe having it more dynamic so that you dont always assume that you only retrieve 2 documents.
Jason
·
Steve H. Are you using a LLM framework such as LangChain or LlamaIndex or are you building your own indexes and queries on top? The spans/traces and Evals might fit your use case Each retrieval action is going to be supported as an LLM span, those spans can have an Eval against them or you can run an Eval at a higher level. We are stitching it together nicely for the LLM frameworks but could be made to work outside of those as well.
Steve H.
·
Jason Im using Llamaindex framework. from llama_index.query_engine import SubQuestionQueryEngine from llama_index.response_synthesizers import get_response_synthesizer query_engine = SubQuestionQueryEngine.from_defaults( query_engine_tools=[ getting_started_tool, community_tool, data_tool, agent_tool, model_tool, query_tool, supporting_tool, tutorials_tool, contributing_tool ], I was referring to Phoenix Data Visualisation rather than tracing. Similar to this :llama_index_search_and_retrieval_tutorial.ipynb
Mikyo
·
query_df = pd.read_parquet( "http://storage.googleapis.com/arize-assets/phoenix/datasets/unstructured/llm/llama-index/arize-docs/query_data_complete3.parquet", ) query_ds = px.Dataset.from_open_inference(query_df) session = px.launch_app(primary=query_ds, corpus=database_ds) The problem is that query_df is based on something that you have already pre-populated and it has a certain structure. When I try to modify the code base so that rather than use query_df I retrofit sample_query_df (which created from me running the queries myself) I get issues and errors.
Steve H. - sorry to hear that - what are the errors you are getting? If you use from_open_inference you have to have the contents conform to open inference traces (https://github.com/Arize-ai/open-inference-spec/blob/main/inference/spec/overview.md)
Mikyo
·
regarding the second issue of complex query engines, in my use case I have multiple indices with each index having its own dedicated tool/query_engine and then having an over arching subqueryengine. I think what I am observing is is returning more than 2 documents, ie) for a given question it will break that down into say 4 sub-queries and each of these sub-queries will retrieve 2 documents, then when we use the classify_relevance function I am not sure it would work properly because I assume it is comparing the original parent query against the documents returned from the sub-queries. Hopefully that makes sense, but to be perfectly honest this seems to be a complex use case. I am just thinking out loud here...but maybe you will need to have some way of getting the relevance of all the sub queries and their correponding retrieved documents. Also maybe having it more dynamic so that you dont always assume that you only retrieve 2 documents.
Steve H. yes you are totally on-point here that it becomes really hard to classify relevance when you have various sub-queries and retrievals. It's partly why we've introduced tracing as we want to support these more complex application architectures. We are definitely going to support these but it is a bit harder. We are building out our evals library and annotating documents as relevant/irrelevant is coming very soon. Unfortunately it's not in the scope of embeddings quite yet. Let me get the team to dogfood the SubQuestionQueryEngine and propose some concrete solutions for you.
Steve H.
·
Mikyo thank you for your response. Regarding the first issue do you have the code or work book as to how you get your data in this format::query_df = pd.read_parquet( "http://storage.googleapis.com/arize-assets/phoenix/datasets/unstructured/llm/llama-index/arize-docs/query_data_complete3.parquet", ) ie) How did you generate all this data? Do you have a code that ca ngenerate this? Regarding the 2nd Issue, yup I agree it is quite complex and a trace would help you visualise what is going on, hopefully you will be able to solve this problem in the near future 🙂

Mikyo
·
For the query_df , is there a way to generate your own query_df from scratch? I am trying to retorfit my "sample_query_df" and replace it with the "query_df". At the moment you just download the parquet file from your google cloud.
I may be mis-understanding your question but of-course! You just need to output the inferences from the callback to file(s) after the queries are run. Are you running your engine in production?
Mikyo
·
1.
I am using LlamaIndex's SubQuestionQueryEngine which effectively breaks down the prompt/question into multiple sub-queries with each sub-query having its own query_engine and hence it returns multiple documents rather than standard 2. Have you thought of what the best way you can handle this use case? Thanks in advance.
I think it depends on how the sub-queries are executed - what are you executing the queries against? Are they the same index / vector store? If so you should be able to accumulate the documents but I can't with 100% confidence say we've tested this flow. Let us know if you hit road blocks.
Mikyo
·
More generically, yes, we are thinking about supporting more complex query engines. We are working on application traces which should show you the traces for each sub-query. https://docs.arize.com/phoenix/concepts/langchain-and-llamaindex-traces
Steve H.
·
Mikyo thanks for your response. Regarding the "sample_query_df" here is a snippet of the code supplied : query_df = pd.read_parquet( "http://storage.googleapis.com/arize-assets/phoenix/datasets/unstructured/llm/llama-index/arize-docs/query_data_complete3.parquet", ) query_ds = px.Dataset.from_open_inference(query_df) session = px.launch_app(primary=query_ds, corpus=database_ds) The problem is that query_df is based on something that you have already pre-populated and it has a certain structure. When I try to modify the code base so that rather than use query_df I retrofit sample_query_df (which created from me running the queries myself) I get issues and errors.
Steve H.
·
Mikyo regarding the second issue of complex query engines, in my use case I have multiple indices with each index having its own dedicated tool/query_engine and then having an over arching subqueryengine. I think what I am observing is is returning more than 2 documents, ie) for a given question it will break that down into say 4 sub-queries and each of these sub-queries will retrieve 2 documents, then when we use the classify_relevance function I am not sure it would work properly because I assume it is comparing the original parent query against the documents returned from the sub-queries. Hopefully that makes sense, but to be perfectly honest this seems to be a complex use case. I am just thinking out loud here...but maybe you will need to have some way of getting the relevance of all the sub queries and their correponding retrieved documents. Also maybe having it more dynamic so that you dont always assume that you only retrieve 2 documents.
Jason
·
Steve H. Are you using a LLM framework such as LangChain or LlamaIndex or are you building your own indexes and queries on top? The spans/traces and Evals might fit your use case Each retrieval action is going to be supported as an LLM span, those spans can have an Eval against them or you can run an Eval at a higher level. We are stitching it together nicely for the LLM frameworks but could be made to work outside of those as well.
Steve H.
·
Jason Im using Llamaindex framework. from llama_index.query_engine import SubQuestionQueryEngine from llama_index.response_synthesizers import get_response_synthesizer query_engine = SubQuestionQueryEngine.from_defaults( query_engine_tools=[ getting_started_tool, community_tool, data_tool, agent_tool, model_tool, query_tool, supporting_tool, tutorials_tool, contributing_tool ], I was referring to Phoenix Data Visualisation rather than tracing. Similar to this :llama_index_search_and_retrieval_tutorial.ipynb
Mikyo
·
query_df = pd.read_parquet( "http://storage.googleapis.com/arize-assets/phoenix/datasets/unstructured/llm/llama-index/arize-docs/query_data_complete3.parquet", ) query_ds = px.Dataset.from_open_inference(query_df) session = px.launch_app(primary=query_ds, corpus=database_ds) The problem is that query_df is based on something that you have already pre-populated and it has a certain structure. When I try to modify the code base so that rather than use query_df I retrofit sample_query_df (which created from me running the queries myself) I get issues and errors.
Steve H. - sorry to hear that - what are the errors you are getting? If you use from_open_inference you have to have the contents conform to open inference traces (https://github.com/Arize-ai/open-inference-spec/blob/main/inference/spec/overview.md)
Mikyo
·
regarding the second issue of complex query engines, in my use case I have multiple indices with each index having its own dedicated tool/query_engine and then having an over arching subqueryengine. I think what I am observing is is returning more than 2 documents, ie) for a given question it will break that down into say 4 sub-queries and each of these sub-queries will retrieve 2 documents, then when we use the classify_relevance function I am not sure it would work properly because I assume it is comparing the original parent query against the documents returned from the sub-queries. Hopefully that makes sense, but to be perfectly honest this seems to be a complex use case. I am just thinking out loud here...but maybe you will need to have some way of getting the relevance of all the sub queries and their correponding retrieved documents. Also maybe having it more dynamic so that you dont always assume that you only retrieve 2 documents.
Steve H. yes you are totally on-point here that it becomes really hard to classify relevance when you have various sub-queries and retrievals. It's partly why we've introduced tracing as we want to support these more complex application architectures. We are definitely going to support these but it is a bit harder. We are building out our evals library and annotating documents as relevant/irrelevant is coming very soon. Unfortunately it's not in the scope of embeddings quite yet. Let me get the team to dogfood the SubQuestionQueryEngine and propose some concrete solutions for you.
Steve H.
·
Mikyo thank you for your response. Regarding the first issue do you have the code or work book as to how you get your data in this format::query_df = pd.read_parquet( "http://storage.googleapis.com/arize-assets/phoenix/datasets/unstructured/llm/llama-index/arize-docs/query_data_complete3.parquet", ) ie) How did you generate all this data? Do you have a code that ca ngenerate this? Regarding the 2nd Issue, yup I agree it is quite complex and a trace would help you visualise what is going on, hopefully you will be able to solve this problem in the near future 🙂