Troubleshooting LLM Integration Issues with Arize's Embedding Navigator

Anouk D. · 2024-06-05T15:18:50.008Z

heyhey, I'm trying to hook up my LLM to Arize and everything is being logged properly, but when I want to check out the embeddings for the prompts and responses I get this error. When logging there is no error being thrown and I can see everything in the UI just fine. Is there something I need to configure separately to get the embedding navigator to work?

15 comments

· Sorted by Oldest

Anouk D.
·
Ah I'm getting an error now. Is it because the vectors are not all of the same length? not sure how I can ensure that they will be all the same length though when the prompts can vary so much 🤔
Kiko C.
·
Hey Anouk D., could you tell me a bit more? Are you trying to trace your application or are you logging inferences of a generative model as you would an ML model? Regardless of the above answer, if you ensure that the model generating the embeddings is the same, the embedding dimensionality will be constant, regardless of the variability of the raw text you're embedding. Give me more details about your setup so I can assist further
Anouk D.
·
I'm simply logging inferences of an LLM like i would for a regular ML model 🙂 I'm fairly new to the whole embedding thing and couldn't find a lot of clear pointers on how to create the embeddings, so I was just sending the tokenized prompt and response vectors. I'm guessing that's not how it works then as they will have variable length 😅 Thanks, I'll have a look then into adjusting how the embeddings are calculated!
Kiko C.
·
Got it. So correct me if I'm wrong on my understanding of your sentence: I was just sending the tokenized prompt and response vectors. So you were taking a sentence and passing the token array (which is basically an array of numbers, each token has a numeric representation) as an embedding vector. Correct?
Kiko C.
·
That's not what an embedding vector is per-se. Or better yet, not how they should be calculated. Of course, if we operate in that way, the embedding dimensionality will depend on the number of tokens, and this is not desirable since you cannot do any math on vectors belonging to different vector spaces (vectors of different dimensionality).
Kiko C.
·
Let me give you some docs and let me know if they help
Kiko C.
·
Our Embeddings documentation is part of our CV guides but they apply equally to NLP or LLMs https://docs.arize.com/arize/computer-vision-cv/how-to-cv
Kiko C.
·
I would suggest starting by reading the "Generate Embeddings" section and the FAQs.
✅1
Anouk D.
·
heyhey, Yes I adjusted my code and I'm using a proper embedding model now to generate the embeddings for the prompt and repsonse. both have consistent dimensions (always 1024). I logged every embedding length before writing it to Arize and it really is all 1024, but the WebApp still keeps telling me that it is inconsistent?
Anouk D.
·
I'm going to share my code here in case I'm missing something obvious. but to me it seems like it should work just fine and the dimensionality shouldn't be an issue. Am I logging it incorrectly?
Anouk D.
·
Ah when I removed the "compared to" row it does work! so nevermind
Kiko C.
·
Hey Anouk D. great! I see you've made a lot of progress. What's happening now is that you're sending embeddings to the same model where the "wrong embeddings" are stored. Thus, Arize sytems see both and you're getting that error. You can do 2 things:
1.
Delete the model llama-3-8b-instruct-arize, and simply send the data again (without the wrong embeddings)
2.
Send the data (without the wrong embeddings) to a new model id
Anouk D.
·
Ah it's because the model still exists of course, I hoped a new version would be enough 🙂 thanks! In the mean time I did manage to get the embedding projections working from the inferences page. Thanks again for your help! my problems are all solved at the moment
Kiko C.
·
Actually a new version should be enough but you need to make sure filters are enabled
Kiko C.
·
Glad you're unblocked!

Anouk D.
·
Ah I'm getting an error now. Is it because the vectors are not all of the same length? not sure how I can ensure that they will be all the same length though when the prompts can vary so much 🤔
Kiko C.
·
Hey Anouk D., could you tell me a bit more? Are you trying to trace your application or are you logging inferences of a generative model as you would an ML model? Regardless of the above answer, if you ensure that the model generating the embeddings is the same, the embedding dimensionality will be constant, regardless of the variability of the raw text you're embedding. Give me more details about your setup so I can assist further
Anouk D.
·
I'm simply logging inferences of an LLM like i would for a regular ML model 🙂 I'm fairly new to the whole embedding thing and couldn't find a lot of clear pointers on how to create the embeddings, so I was just sending the tokenized prompt and response vectors. I'm guessing that's not how it works then as they will have variable length 😅 Thanks, I'll have a look then into adjusting how the embeddings are calculated!
Kiko C.
·
Got it. So correct me if I'm wrong on my understanding of your sentence: I was just sending the tokenized prompt and response vectors. So you were taking a sentence and passing the token array (which is basically an array of numbers, each token has a numeric representation) as an embedding vector. Correct?
Kiko C.
·
That's not what an embedding vector is per-se. Or better yet, not how they should be calculated. Of course, if we operate in that way, the embedding dimensionality will depend on the number of tokens, and this is not desirable since you cannot do any math on vectors belonging to different vector spaces (vectors of different dimensionality).
Kiko C.
·
Let me give you some docs and let me know if they help
Kiko C.
·
Our Embeddings documentation is part of our CV guides but they apply equally to NLP or LLMs https://docs.arize.com/arize/computer-vision-cv/how-to-cv
Kiko C.
·
I would suggest starting by reading the "Generate Embeddings" section and the FAQs.
✅1
Anouk D.
·
heyhey, Yes I adjusted my code and I'm using a proper embedding model now to generate the embeddings for the prompt and repsonse. both have consistent dimensions (always 1024). I logged every embedding length before writing it to Arize and it really is all 1024, but the WebApp still keeps telling me that it is inconsistent?
Anouk D.
·
I'm going to share my code here in case I'm missing something obvious. but to me it seems like it should work just fine and the dimensionality shouldn't be an issue. Am I logging it incorrectly?
Anouk D.
·
Ah when I removed the "compared to" row it does work! so nevermind
Kiko C.
·
Hey Anouk D. great! I see you've made a lot of progress. What's happening now is that you're sending embeddings to the same model where the "wrong embeddings" are stored. Thus, Arize sytems see both and you're getting that error. You can do 2 things:
1.
Delete the model llama-3-8b-instruct-arize, and simply send the data again (without the wrong embeddings)
2.
Send the data (without the wrong embeddings) to a new model id
Anouk D.
·
Ah it's because the model still exists of course, I hoped a new version would be enough 🙂 thanks! In the mean time I did manage to get the embedding projections working from the inferences page. Thanks again for your help! my problems are all solved at the moment
Kiko C.
·
Actually a new version should be enough but you need to make sure filters are enabled
Kiko C.
·
Glad you're unblocked!