Hello, just starting to use Phoenix to improve our prompt engineering. We have datasets from our traces but I'm having a hard time using Dataset Playground to map the user content back into the input of the Playground Prompt template. I've tried {messages[1].content} with no success.
馃挕聽Hint: Mention RunLLM in the thread for followups.
If I have an dataset within an input like:
{
"messages": [
{
"role": "system",
"content": " You are an expert in reading an error messages and how to categorize them. Here is the error:\n"
},
{
"role": "user",
"content": "failed to submit user: 422 {\"errors\":[{\"status\":\"unprocessable_entity\",\"source\":{\"pointer\":\"/data/attributes/all\"},\"detail\":\"invalid DOB\"}]}"
}
]
}How can I have it map example input messages[1]['content'] to the prompt template variable?
馃挕聽Hint: Mention RunLLM in the thread for followups.
Hmm, that's a bummer. It'd be nice if the dataset playground could have more advanced mapping. Either jq, js, or python-like mapping.
Hey Silas B. real human here. I think jq or JSON path makes a lot of sense here. Just to understand your use-case a bit better you are trying to replay messages (e.g. conversations through different LLMs in the playground? Or do you have a single prompt but you want to extract parts of the conversation into the template? In general we have this idea that we have prompt template variables, if these are instrumented properly, the KV pairs get added to the dataset as a flat map. Def. onboard with supporting json path on the variables too.
Silas B. that makes a ton of sense. I filed an issue here for the playground https://github.com/Arize-ai/phoenix/issues/8392 Have you by chance tried the programatic way? https://arize.com/docs/phoenix/datasets-and-experiments/quickstart-datasets You could definitely achieve this via the programatic way. We have support for both python and javascript here. You would just make the task be:
def task(input):
return chat.completions.create(messages=[system_message, ...input.messages])I got to the point where I'm running the experiment, but the experiment fails with this error:
for ex, rep in product(dataset.examples.values(), range(1, repetitions + 1))
^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'list' object has no attribute 'values'I'm loading my dataset via:
from phoenix.client import Client
client = Client()
dataset = client.datasets.get_dataset(dataset="greetings")I'm not entirely sure what the issue is, but I can do dataset.examples and I get a list back.
Hmm, I think I'd need to look at how your dataset got serialized. Let me work with Donny L. and Sanjana Y. to get an example of replaying messages in an experiment https://github.com/Arize-ai/phoenix/issues/8405
Thanks, this is the Python code I used to build the dataset:
import pandas as pd
from phoenix.client import Client
df = pd.read_csv('output.tsv', sep='\t')
client = Client()
dataset = client.datasets.create_dataset(
name="greetings",
dataframe=df,
input_keys=["input"],
output_keys=["output"],
)