Issues with Retrieval Efforts in LLM: Seeking Updated Notebook Solutions

Curt K. · 2024-02-03T23:17:48.338Z

Hey folks - I'm very interested in evaluating my retrieval efforts, but am running into errors with the sample code. This page https://docs.arize.com/phoenix/use-cases/troubleshooting-llm-retrieval-with-vector-stores drives to this Colab notebook: https://colab.research.google.com/github/Arize-ai/phoenix/blob/main/tutorials/langchain_pinecone_search_and_retrieval_tutorial.ipynb which then indicates that before the notebook can be used, this script must be run: https://github.com/Arize-ai/phoenix/blob/main/tutorials/build_arize_docs_index_langchain_pinecone.py. The script fails and throws a number of langchain and pinecone deprecation errors. Is there another notebook I can use that's up to date and focused on this topic? Thanks!

Issues with Retrieval Efforts in LLM: Seeking Updated Notebook Solutions | Arize AI Community

7 comments

· Sorted by Oldest

Mikyo
·
Hey Curt, it looks like you are trying to use embeddings to analyze your retrieval? Thanks for flagging that the notebooks are stale. We’ll look into it! Are you set on using langchain? If not I think the llama index one should be more up to date https://colab.research.google.com/github/Arize-ai/phoenix/blob/main/tutorials/llama_index_search_and_retrieval_tutorial.ipynb

Curt K.

Mikyo and Dat: Just tried the llama index notebook suggested above and it's giving me errors. Is there another notebook related to retrieval that I should be trying? I tried the langchain inference notebooks again and they're still not working as well. At this point neither of the inference notebooks are working, but both of the tracing notebooks ARE working. But it's inference and cluster analysis that I'm really interested in

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-2ef07eb9a729> in <cell line: 11>()
      9 from gcsfs import GCSFileSystem
     10 from IPython.display import YouTubeVideo
---> 11 from llama_index import (
     12     ServiceContext,
     13     StorageContext,

ImportError: cannot import name 'ServiceContext' from 'llama_index' (unknown location)

Curt K.
·
Mikyo and Dat: And lower on this page: https://docs.arize.com/phoenix/use-cases/troubleshooting-llm-retrieval-with-vector-stores there is a link to this notebook: https://colab.research.google.com/github/Arize-ai/phoenix/blob/main/tutorials/llama_index_search_and_retrieval_tutorial.ipynb which also throws errors. I'm really intrigued by your product, but frustratingly, I have yet to find a working notebook that lets me use your cluster analysis interface.
Curt K.
·
and this one doesn't work either: https://colab.research.google.com/github/Arize-ai/phoenix/blob/main/tutorials/evals/evaluate_rag.ipynb#scrollTo=8WvfAuRXFGIX
Mikyo
·
Hey Curt, sorry about that, we had to do a mass migration for llama_index 0.10 and we missed a few. For the pinecone one we have someone updating it but that one I can't give you a concrete ETA on as pinecone changed their SDK drastically. In general Curt, the functionality of Phoenix works - it's just that the interfaces for whatever tool you choose to pull the vectors out have been changing. Thanks for the understanding.
Curt K.
·
awesome. both of the llamaindex notebooks now work. thanks. Regarding the langchain / pinecone notebook... maybe you guys simplify it to use chroma or something similar so you're not reliant on an external api? If the idea of the notebook is to convey how to use langchain and phoenix, maybe pinecone is an unnecessary distraction? (It is for me). Just one man's opinion so take with a grain of salt 🙂
👍1
Mikyo
·
It was a notebook that we presented over a year ago at one of their events but I agree in general - maintaining these notebooks with SDK bindings is a major pain. With our new tracing we can automatically collect the embeddings as the apps run or as the vectors are indexed so there's no need to touch the vector stores themselves. We'll have an end-to-end notebook soon that won't be married to any vector store.
🙌1

Mikyo
·
Hey Curt, it looks like you are trying to use embeddings to analyze your retrieval? Thanks for flagging that the notebooks are stale. We’ll look into it! Are you set on using langchain? If not I think the llama index one should be more up to date https://colab.research.google.com/github/Arize-ai/phoenix/blob/main/tutorials/llama_index_search_and_retrieval_tutorial.ipynb

Curt K.

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-2ef07eb9a729> in <cell line: 11>()
      9 from gcsfs import GCSFileSystem
     10 from IPython.display import YouTubeVideo
---> 11 from llama_index import (
     12     ServiceContext,
     13     StorageContext,

ImportError: cannot import name 'ServiceContext' from 'llama_index' (unknown location)

Curt K.
·
Mikyo and Dat: And lower on this page: https://docs.arize.com/phoenix/use-cases/troubleshooting-llm-retrieval-with-vector-stores there is a link to this notebook: https://colab.research.google.com/github/Arize-ai/phoenix/blob/main/tutorials/llama_index_search_and_retrieval_tutorial.ipynb which also throws errors. I'm really intrigued by your product, but frustratingly, I have yet to find a working notebook that lets me use your cluster analysis interface.
Curt K.
·
and this one doesn't work either: https://colab.research.google.com/github/Arize-ai/phoenix/blob/main/tutorials/evals/evaluate_rag.ipynb#scrollTo=8WvfAuRXFGIX
Mikyo
·
Hey Curt, sorry about that, we had to do a mass migration for llama_index 0.10 and we missed a few. For the pinecone one we have someone updating it but that one I can't give you a concrete ETA on as pinecone changed their SDK drastically. In general Curt, the functionality of Phoenix works - it's just that the interfaces for whatever tool you choose to pull the vectors out have been changing. Thanks for the understanding.
Curt K.
·
awesome. both of the llamaindex notebooks now work. thanks. Regarding the langchain / pinecone notebook... maybe you guys simplify it to use chroma or something similar so you're not reliant on an external api? If the idea of the notebook is to convey how to use langchain and phoenix, maybe pinecone is an unnecessary distraction? (It is for me). Just one man's opinion so take with a grain of salt 🙂
👍1
Mikyo
·
It was a notebook that we presented over a year ago at one of their events but I agree in general - maintaining these notebooks with SDK bindings is a major pain. With our new tracing we can automatically collect the embeddings as the apps run or as the vectors are indexed so there's no need to touch the vector stores themselves. We'll have an end-to-end notebook soon that won't be married to any vector store.
🙌1