Issues with Eval: Hallucination and QA Correctness Logging

·Apr 10, 2024 06:36 AM

Hi Team, has anyone faced any issue in eval where hallucination or qa correctness gets logged as NOT PARSABLE or gets auto changed to from correct/facutal to NOT PARSABLE.

4 comments

· Sorted by Oldest

Xander S.
·
Hey gourav k., this is by design. Even if you provide rails specifying that the output should be either correct or factual, occasionally the LLM will produce unexpected output that cannot be binned into either of those categories. In those cases, the output will be NOT_PARSABLE.
Xander S.
·
If you want to get better visibility into the raw outputs from the LLM, I believe you can pass verbose=True to both llm_classify and run_evals. Hope that helps!
Jason
·
The typical problem is 1 of 2 likely issues:
1.
Using an LLM that is not up to the task, verbose will show the string output, you will see the LLM rambling or off
2.
The template you are using, asks for output values that miss-match the rails. You ask for incorrect / correct but rails are relevant / irrelevant
➕1
gourav k.
·
Thank you for the information. This helps.
👍1

Xander S.
·
Hey gourav k., this is by design. Even if you provide rails specifying that the output should be either correct or factual, occasionally the LLM will produce unexpected output that cannot be binned into either of those categories. In those cases, the output will be NOT_PARSABLE.
Xander S.
·
If you want to get better visibility into the raw outputs from the LLM, I believe you can pass verbose=True to both llm_classify and run_evals. Hope that helps!
Jason
·
The typical problem is 1 of 2 likely issues:
1.
Using an LLM that is not up to the task, verbose will show the string output, you will see the LLM rambling or off
2.
The template you are using, asks for output values that miss-match the rails. You ask for incorrect / correct but rails are relevant / irrelevant
➕1
gourav k.
·
Thank you for the information. This helps.
👍1