Running Experiments with Vector Store Indexes in LlamaIndex
Running Experiments with Pre-existing Vector Store Indexes Hi team! I'm looking for guidance on how to run experiments to evaluate different retrieval modes. Here's my current approach:
- 1.
Create multiple "Project Names" for each tracing and evaluation run. While this works, I believe there might be a more efficient way to do this using the experiment feature.
Is there a sample notebook where I can create an experiment to evaluate different Vector Store indexes in LlamaIndex, for example? Desired Workflow:
- 1.
Create 3 different Vector Store Indexes / Query Engines using 3 different embedding models (OpenAI "ada002", "3-small", "3-large").
- 2.
Pass in a set of 10 test queries with metadata for evaluation.
- 3.
Run the experiment and view the evaluation metrics in the Arize UI and in a dataframe/console in my notebook.
Goal: Rapidly test which configuration gives the best results. Bonus: Many Vector Store Indexes (e.g., Azure AI Search, Elastic, OpenSearch, etc.) have various hyperparameters for the retrieval engine (similar to LLM configurations like temperature, etc.). I wonder if there's a way to perform a hyperparameter sweep to find the best set of query parameters (perhaps even index-time parameters) to automatically optimize these for a given set of test queries. Thanks!
