Steve H.

Mikyo thank you for your response. Regarding the first issue do you have the code or work book as to how you get your data in this format::query_df = pd.read_parquet( "http://storage.googleapis.com/arize-assets/phoenix/datasets/unstructured/llm/llama-index/arize-docs/query_data_complete3.parquet", ) ie) How did you generate all this data? Do you have a code that ca ngenerate this? Regarding the 2nd Issue, yup I agree it is quite complex and a trace would help you visualise what is going on, hopefully you will be able to solve this problem in the near future 🙂

Jason Im using Llamaindex framework. from llama_index.query_engine import SubQuestionQueryEngine from llama_index.response_synthesizers import get_response_synthesizer query_engine = SubQuestionQueryEngine.from_defaults( query_engine_tools=[ getting_started_tool, community_tool, data_tool, agent_tool, model_tool, query_tool, supporting_tool, tutorials_tool, contributing_tool ], I was referring to Phoenix Data Visualisation rather than tracing. Similar to this :llama_index_search_and_retrieval_tutorial.ipynb

Mikyo regarding the second issue of complex query engines, in my use case I have multiple indices with each index having its own dedicated tool/query_engine and then having an over arching subqueryengine. I think what I am observing is is returning more than 2 documents, ie) for a given question it will break that down into say 4 sub-queries and each of these sub-queries will retrieve 2 documents, then when we use the classify_relevance function I am not sure it would work properly because I assume it is comparing the original parent query against the documents returned from the sub-queries. Hopefully that makes sense, but to be perfectly honest this seems to be a complex use case. I am just thinking out loud here...but maybe you will need to have some way of getting the relevance of all the sub queries and their correponding retrieved documents. Also maybe having it more dynamic so that you dont always assume that you only retrieve 2 documents.

Mikyo thanks for your response. Regarding the "sample_query_df" here is a snippet of the code supplied : query_df = pd.read_parquet( "http://storage.googleapis.com/arize-assets/phoenix/datasets/unstructured/llm/llama-index/arize-docs/query_data_complete3.parquet", ) query_ds = px.Dataset.from_open_inference(query_df) session = px.launch_app(primary=query_ds, corpus=database_ds) The problem is that query_df is based on something that you have already pre-populated and it has a certain structure. When I try to modify the code base so that rather than use query_df I retrofit sample_query_df (which created from me running the queries myself) I get issues and errors.

Xander S. I was thinking to use Llama 2 7B chat model or other variants of this LLM. Maybe even finetune an opensource model specifically for evaluation. I think this would be greatly particularly if you have lots of data and queries.

Mikyo Perfect thanks heaps for that. Keep up the great work, loving what Im seeing thus far.

Thanks Mikyo Xander S. Jason for your quick response. The code I was following was: https://colab.research.google.com/github/Arize-ai/phoenix/blob/main/tutorials/llama_index_search_and_retrieval_tutorial.ipynb . I was able to successfully use an opensource LLM for the sythnesis. I think for the embedding model it is just because the sample data that is pre-index has vector of different length than say an open source one from Huggingface. The main thing I guess for the "classify_relevance(query_text, document_text, evals_model_name)" function it only accepts GPT3.5 or GPT-4. Is there a way to change it t use saying Llamaindex,Langchain or Huggingface frame work?