Sorry for the xpost. Should’ve posted this here.

· Sorted by Oldest

Roger Y.
·
·
That feature isn’t supported at the moment. Would you mind filing an issue so we can track it? cc. Elizabeth H. Dustin N.
Andrew S.
·
·
since littlellm can be used under the hood, shouldn’t it be possible now?
Andrew S.
·
·
I’m going to try now
Roger Y.
·
·
that’s a good point. i was not aware of it.
Andrew S.
·
·
afaict, that would entail subclassing class LLM(*, provider=None, model=None, client=None, initial_per_second_request_rate=None, **kwargs) ? 🤔 or this:
Alternative: Fully Custom LLM Evaluator Alternatively, for LLM-as-a-judge tasks that don’t fit the classification paradigm, it is also possible to create a custom evaluator that implements the base LLMEvaluator class. We can implement our own LLMEvaluator for almost any complex eval that doesn’t fit into the classification type.

Roger Y.
·
·
That feature isn’t supported at the moment. Would you mind filing an issue so we can track it? cc. Elizabeth H. Dustin N.
Andrew S.
·
·
since littlellm can be used under the hood, shouldn’t it be possible now?
Andrew S.
·
·
I’m going to try now
Roger Y.
·
·
that’s a good point. i was not aware of it.
Andrew S.
·
·
afaict, that would entail subclassing class LLM(*, provider=None, model=None, client=None, initial_per_second_request_rate=None, **kwargs) ? 🤔 or this:
Alternative: Fully Custom LLM Evaluator Alternatively, for LLM-as-a-judge tasks that don’t fit the classification paradigm, it is also possible to create a custom evaluator that implements the base LLMEvaluator class. We can implement our own LLMEvaluator for almost any complex eval that doesn’t fit into the classification type.