Using llmclassify
Prompt ends like this:
[BEGIN DATA]
Ruby Script:
{{{ruby_script}}}
Typescript Script:
{{{ts_script}}}
[END DATA]
Please analyze the data carefully and provide an explanation followed by your response.
EXPLANATION: Provide your reasoning step by step, evaluating why the scripts are functionally the same or not.
LABEL: "correct" or "incorrect"
"""Template:
template = ClassificationTemplate(
rails = ['correct', 'incorrect'],
template = LOGIC_LLM_JUDGE_PROMPT,
delimiters = ("{{{", "}}}"),Eval:
with suppress_tracing():
logic_eval = llm_classify(
dataframe = code_gen_df,
template = template,
rails = ['correct', 'incorrect'],
model=omodel,
provide_explanation=True,
include_prompt=True,
verbose=True,
)
