Elizabeth H.

Hi Ourahou O., are you working in python or typescript? I can link some resources. I'll also DM you with a link to sign up for my evals office hours

Truong H. right now the best way is to use the _dumps method, like you mentioned. We will look into adding a utility for more graceful conversion, though!

Kyle J. this is also likely related to how the LLM response is parsed when not using tool-calling. You should be able to set use_function_calling_if_available=True when you run llm_classify with your evaluator. What model are you using?

Nico G., have you tried increasing the timeout from the default?

In the typescript library, we delegate to the AI SDK from Vercel, but there isn't one clear "winner" on the python side.

You're the first person we've heard of using this llm tool. We decided to start with adaptors to the 3 most common LLM clients we see people using: openai, langchain, and litellm. Between those 3, pretty much every model is supported. We intentionally avoided strict dependency on any one package, such that users can choose which SDK they prefer to use.

Just a heads up, though, that we are still working on supporting LLM invocation parameters with evals 2.0 so you won't be able to specify max_tokens or max_completion_tokens right now. We are actively working on this now so it should be available soon. cc: 🔒[private user], 🔒[private user]

We are also working on the docs for the LLM abstraction. We limited provider support to anthropic and openai initially because we haven't tested others, but we can theoretically expand to support any provider compatible with one of openai, langchain, or litellm as client SDKs. There are two places where you can pass additional kwargs - at LLM instantiation you should pass any "client" args like api_base . You can pass invocation params like temperature to the LLM methods, but we haven't set up the LLMEvaluator/ClassificationEvaluator to pass through kwargs yet. We are getting to it soon.