Hmm, not sure if I understood that correctly. But I'll try explain my side better.
Today I have a separate project just to run evals against a llama index application.
We don't run evals in notebooks, we have plain scripts that integrate with phoenix client and phoenix evals
Then we have something like this:
results = phoenix.evals.llm_classify(
dataframe=df,
template=template,
model=phoenix.evals.OpenAIModel(model=MODEL_NAME, temperature=TEMPERATURE, api_key=API_KEY),
rails=rails,
verbose=False,
provide_explanation=provide_explanation,
progress_bar_format=None,
)
I assume that for executing the evals, the phoenix.evals.OpenAIModel is probably being used. My question is: is this OpenAiModel instrumented? I like to see LLM calls directly on phoenix, I would like to see a call in there for, for example, the HALLUCINATION_SPAN_PROMPT_BASE_TEMPLATE filled with it's parameters system_message, user_message and output
