Evaluating Models in VertexAI: Seeking Advice on Alternatives

Roméo I. · 2024-09-10T13:40:24.143Z

Hi everyone, it's me again ! I'm trying to run evaluations on my spans, I have to use VertexAi as the evaluation model. I used the class VertexAIModel to instantiate my model, but I the only model I manged to make it work was "publishers/google/models/text-bison". But ultimately I want to use a better model such as "publishers/google/models/gemini-1.5-pro-001" or even better "publishers/anthropic/models/claude-3-5-sonnet". Am I missing something obvious to make it work or is there or work around like overriding the VertexAIModel class ??

15 comments

· Sorted by Oldest

RunLLM
APP
·
The VertexAIModel class in Phoenix supports Google's Vertex AI models, but it appears that the specific models you mentioned ("gemini-1.5-pro-001" and "claude-3-5-sonnet") are not directly supported by the VertexAIModel class. The GeminiModel class is designed for Google's Gemini models and might be more appropriate for "gemini-1.5-pro-001".For "claude-3-5-sonnet", which is an Anthropic model, you would need to use the AnthropicModel class instead of VertexAIModel.Would you like more details on how to instantiate and use these specific model classes?Sources: - evals.models.vertex
💡 Hint: Mention RunLLM in the thread for followups.
Roméo I.
·
The thing I dont understand is why there is restrictions on the final model used behind the vertexai api
Roger Y.
·
Dustin N. do you know the answer offhand?
Dustin N.
·
the API used to access older Vertex models and the Gemini models was completely different, so we made a new wrapper instead of having the old one need to know which models used what interface
1
Roger Y.
·
Roméo I. are you using GeminiModel?
Roméo I.
·
Yes sorry Roger Y., I continued my investigation and found this class, so it works for the Gemini models which is great, but is it possible to use the anthropics models from vertexai ?
Roger Y.
·
we also have AnthropicModel, would that work for you?
Roméo I.
·
no I'm bind to vertexai, it's mandatory for this project and the the user datas behind it
Roger Y.
·
so you need this, correct?
from anthropic import AnthropicVertex
👍1
Roger Y.
·
let me do some digging on it and get back to you

Roger Y.

Roméo I. can you give the following a try? i was able to connect to vertex using it, but i don’t have anthropic access specifically on vertex.

from phoenix.evals import AnthropicModel

class AnthropicVertexModel(AnthropicModel):
    def _init_client(self) -> None:
        import anthropic

        self._anthropic = anthropic
        self.client = self._anthropic.AnthropicVertex()
        self.async_client = self._anthropic.AsyncAnthropicVertex()

eval_model = AnthropicVertexModel()

Roger Y.
·
if it’s working for you we can add it to the code base
Roméo I.
·
Terrific, it works ! For a proper implementation you'll just need to be able to provide the project_id, the location, the model name etc...
Roméo I.
·
Thank you very much Roger Y., it's always a pleasure tu work with Phoenix, you guys are so helpful and reactive!!
Roger Y.
·
np. always happy to assist

RunLLM
APP
·
The VertexAIModel class in Phoenix supports Google's Vertex AI models, but it appears that the specific models you mentioned ("gemini-1.5-pro-001" and "claude-3-5-sonnet") are not directly supported by the VertexAIModel class. The GeminiModel class is designed for Google's Gemini models and might be more appropriate for "gemini-1.5-pro-001".For "claude-3-5-sonnet", which is an Anthropic model, you would need to use the AnthropicModel class instead of VertexAIModel.Would you like more details on how to instantiate and use these specific model classes?Sources: - evals.models.vertex
💡 Hint: Mention RunLLM in the thread for followups.
Roméo I.
·
The thing I dont understand is why there is restrictions on the final model used behind the vertexai api
Roger Y.
·
Dustin N. do you know the answer offhand?
Dustin N.
·
the API used to access older Vertex models and the Gemini models was completely different, so we made a new wrapper instead of having the old one need to know which models used what interface
1
Roger Y.
·
Roméo I. are you using GeminiModel?
Roméo I.
·
Yes sorry Roger Y., I continued my investigation and found this class, so it works for the Gemini models which is great, but is it possible to use the anthropics models from vertexai ?
Roger Y.
·
we also have AnthropicModel, would that work for you?
Roméo I.
·
no I'm bind to vertexai, it's mandatory for this project and the the user datas behind it
Roger Y.
·
so you need this, correct?
from anthropic import AnthropicVertex
👍1
Roger Y.
·
let me do some digging on it and get back to you

Roger Y.

Roméo I. can you give the following a try? i was able to connect to vertex using it, but i don’t have anthropic access specifically on vertex.

from phoenix.evals import AnthropicModel

class AnthropicVertexModel(AnthropicModel):
    def _init_client(self) -> None:
        import anthropic

        self._anthropic = anthropic
        self.client = self._anthropic.AnthropicVertex()
        self.async_client = self._anthropic.AsyncAnthropicVertex()

eval_model = AnthropicVertexModel()

Roger Y.
·
if it’s working for you we can add it to the code base
Roméo I.
·
Terrific, it works ! For a proper implementation you'll just need to be able to provide the project_id, the location, the model name etc...
Roméo I.
·
Thank you very much Roger Y., it's always a pleasure tu work with Phoenix, you guys are so helpful and reactive!!
Roger Y.
·
np. always happy to assist