Seeking Solutions for Visual Content in Knowledge Base

Ou J. · 2023-09-17T23:32:14.685Z

Dear all, I am new to here, just watched couple of Youtube video. your products are great, that's the solution many companies are looking for. I have one simple maybe stupid question. For my company knowledge base, we have many documents which have instructions with pictures. It seems that all those text contents can be answered well, but we are also looking for one solution to show all relative picture, that will be much helpful. Does Arize platform or Phonenix can help on this ? or Do you have any suggestion? Really appreicate for your answer.

Seeking Solutions for Visual Content in Knowledge Base | Arize AI Community

5 comments

· Sorted by Oldest

Jason
·
Ou J. In terms of a Q&A retrieval for question and answer purposes, that then leverages the pictures as part of the process. It could be done put probably slightly cutting edge (haven't seen too many people do that yet) with more work versus out of the box solutions for pure text. As a note, with Gemeni coming from Google and OpenAI likely launching a multi-modal model soon (image + text), this use case is likely to be the hot one over the next 12 months. In order to do this today you need to choose an embedding that can be multi-modal. I've seen people do this with certain CLIP models. You could return images similar to your text but can't have the LLM respond just yet based on the image.
Ou J.
·
I am deeply appreciative of your swift response, Jason The requirement here is not to create fresh imagery through the LLM, but rather to display existing images already embedded within our documentation. In your opinion, might there exist a more straightforward approach to achieve this objective rather than anticipating the unveiling of the multi-model model? thanks again
Jason
·
It really depends on the use case, assuming a search and retrieval, there are a lot of things you could build. If you want to return images similar to a query you could use a joint embedding model such as CLIP (text and image) with vector db. Return similar images along with text. You could even do something simpler such as, just keep image links in the text chunks they are related to, display images related to the chunk when it is returned (probably easier with todays stack). Might need to test a couple document loaders/approaches to do this. The coming multi-modal models will allow you to use both images and text to generate text responses (not just image gen), they understand textually what is in the images. I think the retrieval stacks will also be reworked for better for multi-modal support
Jason
·
Jerry L. Any good data loaders that allow you to keep/extract structured image url and store in vector db along with chunk?
Ou J.
·
Thanks very much Jason for your answer, it helps me a lot, I will try to start with option 2 by keeping the links in the chunk. Jason Have a great day.

Jason
·
Ou J. In terms of a Q&A retrieval for question and answer purposes, that then leverages the pictures as part of the process. It could be done put probably slightly cutting edge (haven't seen too many people do that yet) with more work versus out of the box solutions for pure text. As a note, with Gemeni coming from Google and OpenAI likely launching a multi-modal model soon (image + text), this use case is likely to be the hot one over the next 12 months. In order to do this today you need to choose an embedding that can be multi-modal. I've seen people do this with certain CLIP models. You could return images similar to your text but can't have the LLM respond just yet based on the image.
Ou J.
·
I am deeply appreciative of your swift response, Jason The requirement here is not to create fresh imagery through the LLM, but rather to display existing images already embedded within our documentation. In your opinion, might there exist a more straightforward approach to achieve this objective rather than anticipating the unveiling of the multi-model model? thanks again
Jason
·
It really depends on the use case, assuming a search and retrieval, there are a lot of things you could build. If you want to return images similar to a query you could use a joint embedding model such as CLIP (text and image) with vector db. Return similar images along with text. You could even do something simpler such as, just keep image links in the text chunks they are related to, display images related to the chunk when it is returned (probably easier with todays stack). Might need to test a couple document loaders/approaches to do this. The coming multi-modal models will allow you to use both images and text to generate text responses (not just image gen), they understand textually what is in the images. I think the retrieval stacks will also be reworked for better for multi-modal support
Jason
·
Jerry L. Any good data loaders that allow you to keep/extract structured image url and store in vector db along with chunk?
Ou J.
·
Thanks very much Jason for your answer, it helps me a lot, I will try to start with option 2 by keeping the links in the chunk. Jason Have a great day.