Seeking Help with Hallucination LLM Judge Type Error in Python

James G. · 2024-11-05T07:13:59.086Z

Hello team, good to meet y'all, James here from Singapore. I'm trying out the usage example for rag-llm-prompt-evaluator-guard - hallucination LLM judge. To try it out, I'm using the server approach, not SDK. With the config.py file containing the guard set up, I tried running the .validate function in a separate script but faced an error mentioning that there's a type error for the HallucinationPrompt object(Error in the pictures attached). I'm confuse as in the terminal output, it shows that my guard: hallucination_free is active, but gets this error when I'm trying to validate. Can anyone advise? It would be greatly appreciated 🙂 Please excuse me if there's a mistake on my part, I'm very new to this and would like to seek the community's help Link to repo here: https://github.com/Arize-ai/rag-llm-prompt-evaluator-guard

Seeking Help with Hallucination LLM Judge Type Error in Python | Arize AI Community

3 comments

· Sorted by Oldest

John G.
·
Hey James! That repo is dependent on a 3rd party, and isn't something we're maintaining consistently. It's possible our code is out of date I should be able to get you something more up to date. That repo does hallucination checks online during execution of the application. Are you trying to do that, or just trying to do hallucination evals in general?
James G.
·
Thanks for getting back to me John G., I’m mainly need to get more info for during execution. But I’m open to see the hallucination evals in case that might fit my use case.

John G.

Gotcha - if you're wanting to run the evals live in your application on each response, you can replicate the guardrails behavior and still use our evaluator fairly easily. Here's what that looks like:

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from phoenix.evals import HallucinationEvaluator, OpenAIModel

def build_pipeline():
    documents = SimpleDirectoryReader("paul_graham").load_data()
    index = VectorStoreIndex.from_documents(documents)
    query_engine = index.as_query_engine()
    return query_engine

def validate_result(query, result):
    evaluator = HallucinationEvaluator(OpenAIModel("gpt-4o-mini"))
    record = {
        "input": query,
        "output": result.response,
        "reference": [doc.node.text for doc in result.source_nodes]
    }
    eval_result = evaluator.evaluate(record)
    return eval_result[0]

if __name__ == "__main__":
    query = "What is Brian Chesky's nickname?"
    
    query_engine = build_pipeline()
    result = query_engine.query(query)
    
    if validate_result(query, result) != "factual":
        print("I can't answer that")
    else:
        print(result)

To run that exact code, run export OPENAI_API_KEY='your api key' or add it in a .env file, and create a local directory of text files called paul_graham/. Lmk if you run into issues! You might also want to check out Online Tasks in Arize. These don't block the responses in your application, but allow you to add hallucination and other labels to your app as it's executing

John G.
·
Hey James! That repo is dependent on a 3rd party, and isn't something we're maintaining consistently. It's possible our code is out of date I should be able to get you something more up to date. That repo does hallucination checks online during execution of the application. Are you trying to do that, or just trying to do hallucination evals in general?
James G.
·
Thanks for getting back to me John G., I’m mainly need to get more info for during execution. But I’m open to see the hallucination evals in case that might fit my use case.

John G.

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from phoenix.evals import HallucinationEvaluator, OpenAIModel

def build_pipeline():
    documents = SimpleDirectoryReader("paul_graham").load_data()
    index = VectorStoreIndex.from_documents(documents)
    query_engine = index.as_query_engine()
    return query_engine

def validate_result(query, result):
    evaluator = HallucinationEvaluator(OpenAIModel("gpt-4o-mini"))
    record = {
        "input": query,
        "output": result.response,
        "reference": [doc.node.text for doc in result.source_nodes]
    }
    eval_result = evaluator.evaluate(record)
    return eval_result[0]

if __name__ == "__main__":
    query = "What is Brian Chesky's nickname?"
    
    query_engine = build_pipeline()
    result = query_engine.query(query)
    
    if validate_result(query, result) != "factual":
        print("I can't answer that")
    else:
        print(result)