Hi all, just joined the slack. I recently started watching the Arize videos/webinars on youtube and have been learning a LOT so big thank you to the Arize team for the amazing content!
As I was watching one of the videos, a thought came up. I recall Xander talking about a common problem where the context you get back from your vector database is not actually relevant to (answering) the question at hand (similar != relevant).
Here’s my idea: Training a model to determine when the given context (from a retrieved chunk) is relevant/partially relevant to answer the original query. I say “partially relevant” because a lot of times the entire answer cannot be found in ONE chunk alone, but rather the context needed is distributed across several chunks. This could be binary classification where the model determines the chunk as either “relevant” or “not relevant”. In that case, we can essentially “filter out” all the irrelevant context. Alternatively, we could classify on some relevancy scale, and have some sort of threshold, or ranking chunks based on this new metric, etc…
And I suppose all this would be to replace the LLM-assisted ranking approach using GPT-4 that Xander discussed, and especially help cut down on those open ai costs.
I’m wondering has anyone tried this approach out before and/or have any idea of how well this works?
