Hello Team, I'm trying to use Claude Sonnet 3.5 via LiteLLM. So far the only tutorial I found is regarding ollama running locally. How would one define the LiteLLM and the RelevanceEvaluator to point to a different machine running the litellm endpoint? Basically the content from here: https://docs.arize.com/phoenix/api/evaluation-models#litellmmodel
Hey Teodor, you get me again. haha. So our model classes are largely passthrough to the SDKs with concurrency patterns built in. Curious before I dive in on the LiteLLM stuff why you may have chosen LiteLLM over the native anthropic model class? https://docs.arize.com/phoenix/api/evaluation-models#anthropicmodel
But yeah if you want to use lightly you definitely can! https://litellm.vercel.app/docs/providers/anthropic
we use litellm with presidio for pii masking :)
Oh gotcha. Yeah our LiteLLM model class is a pretty light wrapper but it should pass through anything you may need I hope:
def _generate(self, prompt: str, **kwargs: Dict[str, Any]) -> str:
messages = self._get_messages_from_prompt(prompt)
response = self._litellm.completion(
model=self.model,
messages=messages,
temperature=self.temperature,
max_tokens=self.max_tokens,
top_p=self.top_p,
num_retries=self.num_retries,
request_timeout=self.request_timeout,
**self.model_kwargs,
)
return str(response.choices[0].message.content)So it maps 1 to 1 with the litellm invocation - you just have to pass whatever you need in model_kwargs Let me know if I'm maybe misunderstanding your question 馃槄
so i guess the question is more around where do you set the hostname for the litellm machine. for example if you use ollama you set an env variable, is it the same variable ?
Not super familiar with litellm with claude tbh. But it looks like it's inconsistent and is a parameter? https://github.com/BerriAI/litellm/blob/2bd8205ef56e25852a0bca1e9aaff4dacf950356/litellm/llms/anthropic.py#L457
which means you would put it in model_kwargs
Yeah lmk how it goes. Last we checked the anthropic models they were a bit chatty 馃檪
