Hello, just starting to use Phoenix to improve our prompt engineering. We have datasets from our traces but I'm having a hard time using Dataset Playground to map the user content back into the input of the Playground Prompt template. I've tried {messages[1].content} with no success.
In Phoenix's Prompt Playground, you should use template variables (e.g., {{question}} for mustache or {question} for f-string) that match the column names in your dataset. When running prompts over a dataset, Phoenix automatically maps dataset columns to these variables in your prompt template鈥攕o if your dataset has a column named user_content, your prompt should use {{user_content}} (mustache) or {user_content} (f-string) as the variable in the template. Direct references like {messages[1].content} are not supported in prompt templates [source]聽[source].Would you like more detail or a code example for your specific dataset structure?Sources:
- Quickstart: Prompts (UI) | Phoenix
馃挕聽Hint: Mention RunLLM in the thread for followups.
If I have an dataset within an input like:
{
"messages": [
{
"role": "system",
"content": " You are an expert in reading an error messages and how to categorize them. Here is the error:\n"
},
{
"role": "user",
"content": "failed to submit user: 422 {\"errors\":[{\"status\":\"unprocessable_entity\",\"source\":{\"pointer\":\"/data/attributes/all\"},\"detail\":\"invalid DOB\"}]}"
}
]
}How can I have it map example input messages[1]['content'] to the prompt template variable?
To map messages[1]['content'] from your dataset to a prompt template variable in Phoenix, you should preprocess your dataset so that the user message content is extracted into its own column (e.g., user_content). Then, reference that variable in your prompt template as {{user_content}} (mustache) or {user_content} (f-string). Phoenix does not support direct indexing like messages[1]['content'] in prompt templates [source]聽[source].Would you like a step-by-step example of how to preprocess your dataset and set up the prompt template?Sources:
- https://github.com/Arize-ai/phoenix/blob/main/app/src/pages/playground/tests/playgroundUtils.test.ts
- https://github.com/Arize-ai/phoenix/pull/8130
馃挕聽Hint: Mention RunLLM in the thread for followups.
Hmm, that's a bummer. It'd be nice if the dataset playground could have more advanced mapping. Either jq, js, or python-like mapping.
Hey Silas B. real human here. I think jq or JSON path makes a lot of sense here. Just to understand your use-case a bit better you are trying to replay messages (e.g. conversations through different LLMs in the playground? Or do you have a single prompt but you want to extract parts of the conversation into the template? In general we have this idea that we have prompt template variables, if these are instrumented properly, the KV pairs get added to the dataset as a flat map. Def. onboard with supporting json path on the variables too.
Silas B. that makes a ton of sense. I filed an issue here for the playground https://github.com/Arize-ai/phoenix/issues/8392 Have you by chance tried the programatic way? https://arize.com/docs/phoenix/datasets-and-experiments/quickstart-datasets You could definitely achieve this via the programatic way. We have support for both python and javascript here. You would just make the task be:
def task(input):
return chat.completions.create(messages=[system_message, ...input.messages])I got to the point where I'm running the experiment, but the experiment fails with this error:
for ex, rep in product(dataset.examples.values(), range(1, repetitions + 1))
^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'list' object has no attribute 'values'I'm loading my dataset via:
from phoenix.client import Client
client = Client()
dataset = client.datasets.get_dataset(dataset="greetings")I'm not entirely sure what the issue is, but I can do dataset.examples and I get a list back.
Hmm, I think I'd need to look at how your dataset got serialized. Let me work with Donny L. and Sanjana Y. to get an example of replaying messages in an experiment https://github.com/Arize-ai/phoenix/issues/8405
Thanks, this is the Python code I used to build the dataset:
import pandas as pd
from phoenix.client import Client
df = pd.read_csv('output.tsv', sep='\t')
client = Client()
dataset = client.datasets.create_dataset(
name="greetings",
dataframe=df,
input_keys=["input"],
output_keys=["output"],
)