Unexpected Issues with LLM Classify API and Function Calling

Hi -- We are observing unexpected results for relevancy evaluation using the llm_classify API with our local LLM interacting via the OpenAI interface (endpoints). We have seen the Assistant API feature (function calling) of the chat completion request is being used for relevancy inference. functions=[{'name': 'record_response', 'description': 'A function to record your response.', 'parameters': {'type': 'object', 'properties': {'explanation': {'type': 'string', 'description': 'Explanation of the reasoning for your response.'}, 'response': {'type': 'string', 'description': 'Your response.', 'enum': ['relevant', 'unrelated']}}, 'required': ['explanation', 'response'] } }] function_call={'name': 'record_response'} Our local LLM doesn't support function calling but returns the expected response via message/content: 'message': {'role': 'assistant', 'content': ' EXPLANATION: The reference text describes a process ..... Therefore, the label is \'relevant\'. \nLABEL: "relevant"'} However, llm_classify is not able to parse this message and properly extract the explanation/label, even though it's in the same format as instructed to the LLM via the prompt. Is this a known issue? Thanks, David