Evaluating Models in VertexAI: Seeking Advice on Alternatives

Roméo I. · 2024-09-10T13:40:24.143Z

Hi everyone, it's me again ! I'm trying to run evaluations on my spans, I have to use VertexAi as the evaluation model. I used the class VertexAIModel to instantiate my model, but I the only model I manged to make it work was "publishers/google/models/text-bison". But ultimately I want to use a better model such as "publishers/google/models/gemini-1.5-pro-001" or even better "publishers/anthropic/models/claude-3-5-sonnet". Am I missing something obvious to make it work or is there or work around like overriding the VertexAIModel class ??

Evaluating Models in VertexAI: Seeking Advice on Alternatives | Arize AI Community

15 comments

· Sorted by Oldest

RunLLM
APP
·
The VertexAIModel class in Phoenix supports Google's Vertex AI models, but it appears that the specific models you mentioned ("gemini-1.5-pro-001" and "claude-3-5-sonnet") are not directly supported by the VertexAIModel class. The GeminiModel class is designed for Google's Gemini models and might be more appropriate for "gemini-1.5-pro-001".
For "claude-3-5-sonnet", which is an Anthropic model, you would need to use the AnthropicModel class instead of VertexAIModel.
Would you like more details on how to instantiate and use these specific model classes?
Sources: - evals.models.vertex
💡 Hint: Mention RunLLM in the thread for followups.
Roméo I.
·
The thing I dont understand is why there is restrictions on the final model used behind the vertexai api
Roger Y.
·
Dustin N. do you know the answer offhand?
Dustin N.
·
the API used to access older Vertex models and the Gemini models was completely different, so we made a new wrapper instead of having the old one need to know which models used what interface
1
Roger Y.
·
Roméo I. are you using GeminiModel?
Roméo I.
·
Yes sorry Roger Y., I continued my investigation and found this class, so it works for the Gemini models which is great, but is it possible to use the anthropics models from vertexai ?
Roger Y.
·
we also have AnthropicModel, would that work for you?
Roméo I.
·
no I'm bind to vertexai, it's mandatory for this project and the the user datas behind it
Roger Y.
·
so you need this, correct?
from anthropic import AnthropicVertex
👍1
Roger Y.
·
let me do some digging on it and get back to you

Roger Y.

Roméo I. can you give the following a try? i was able to connect to vertex using it, but i don’t have anthropic access specifically on vertex.

from phoenix.evals import AnthropicModel

class AnthropicVertexModel(AnthropicModel):
    def _init_client(self) -> None:
        import anthropic

        self._anthropic = anthropic
        self.client = self._anthropic.AnthropicVertex()
        self.async_client = self._anthropic.AsyncAnthropicVertex()

eval_model = AnthropicVertexModel()

Roger Y.
·
if it’s working for you we can add it to the code base
Roméo I.
·
Terrific, it works ! For a proper implementation you'll just need to be able to provide the project_id, the location, the model name etc...
Roméo I.
·
Thank you very much Roger Y., it's always a pleasure tu work with Phoenix, you guys are so helpful and reactive!!
Roger Y.
·
np. always happy to assist

RunLLM
APP
·
The VertexAIModel class in Phoenix supports Google's Vertex AI models, but it appears that the specific models you mentioned ("gemini-1.5-pro-001" and "claude-3-5-sonnet") are not directly supported by the VertexAIModel class. The GeminiModel class is designed for Google's Gemini models and might be more appropriate for "gemini-1.5-pro-001".
For "claude-3-5-sonnet", which is an Anthropic model, you would need to use the AnthropicModel class instead of VertexAIModel.
Would you like more details on how to instantiate and use these specific model classes?
Sources: - evals.models.vertex
💡 Hint: Mention RunLLM in the thread for followups.
Roméo I.
·
The thing I dont understand is why there is restrictions on the final model used behind the vertexai api
Roger Y.
·
Dustin N. do you know the answer offhand?
Dustin N.
·
the API used to access older Vertex models and the Gemini models was completely different, so we made a new wrapper instead of having the old one need to know which models used what interface
1
Roger Y.
·
Roméo I. are you using GeminiModel?
Roméo I.
·
Yes sorry Roger Y., I continued my investigation and found this class, so it works for the Gemini models which is great, but is it possible to use the anthropics models from vertexai ?
Roger Y.
·
we also have AnthropicModel, would that work for you?
Roméo I.
·
no I'm bind to vertexai, it's mandatory for this project and the the user datas behind it
Roger Y.
·
so you need this, correct?
from anthropic import AnthropicVertex
👍1
Roger Y.
·
let me do some digging on it and get back to you

Roger Y.

Roméo I. can you give the following a try? i was able to connect to vertex using it, but i don’t have anthropic access specifically on vertex.

from phoenix.evals import AnthropicModel

class AnthropicVertexModel(AnthropicModel):
    def _init_client(self) -> None:
        import anthropic

        self._anthropic = anthropic
        self.client = self._anthropic.AnthropicVertex()
        self.async_client = self._anthropic.AsyncAnthropicVertex()

eval_model = AnthropicVertexModel()

Roger Y.
·
if it’s working for you we can add it to the code base
Roméo I.
·
Terrific, it works ! For a proper implementation you'll just need to be able to provide the project_id, the location, the model name etc...
Roméo I.
·
Thank you very much Roger Y., it's always a pleasure tu work with Phoenix, you guys are so helpful and reactive!!
Roger Y.
·
np. always happy to assist