How to Format Code for llm_classify Prompts

How to Format Code for llm_classify Prompts | Arize AI Community

RunLLM
APP
·
It seems likely that you are trying to include code snippets within a prompt using Mustache formatting, which uses {{ and }} for placeholders. To escape these characters, you can use triple braces {{{ and }}} or escape them with a backslash. However, the specific method for escaping may depend on the context in which you are using the prompt.If you are using Phoenix, you can format your prompt using Mustache or F-string formats. For Mustache, you can escape variables by using triple braces or by adding a backslash before the braces. For example, to include a code snippet, you might use: LBKS{{{code_snippet}}}LBKSWould you like more details on how to implement this in your specific use case?Sources: - Prompt Format
💡 Hint: Mention RunLLM in the thread for followups.
Xander S.
·
Hey Aditya A., I take it the issue is that your code contains braces?

·

Tried the following: Prompt:

lorem ipsum

ruby script:
{{{ruby}}}

typescript script:
{{{typescript}}}

Where

ruby = """ 
'''ruby
code 
'''
"""

typescript = """
'''typescript
code 
'''
"""

prompt.replace("{ruby}", ruby).replace("{typescript}", typescript)

Does not help

Xander S.
·
Can you use single braces?

Aditya T.

·

Like:

lorem ipsum

ruby script:
{ruby}

typescript script:
{typescript}

Aditya T.
·
Does not work hitting the same issue as before:
Retries exhausted after 1 attempts: Missing template variable: ' report '
This I believe is caused due to this line in typescript code:
import { report } from 'mylib';
Xander S.
·
This is inelegant, but can you try escaping braces inside your TS code. So
import {{ report }} from 'mylib';
Xander S.
·
The other thing you can try is the template parameter to llm_classify takes in a delimiters argument. You can pass custom delimiters such as ("{{{", "}}}")
Aditya T.
·
Will passing the prompt as a markdown work?
Aditya T.
·
The problem with {{ approach is it becomes unwieldy dealing with nested code blocks
Xander S.
·
I hear you. Can you try to custom delimiters?

Aditya T.

·

Like so?

    logic_eval = llm_classify(
        dataframe = code_gen_df,
        template = LOGIC_LLM_JUDGE_PROMPT.replace("{ruby}", ruby).replace("{typescript}", typescript),
        rails = ['logically_correct', 'logically_correct'],
        model=LiteLLMModel(model="ollama/llama3.2:latest"),
        provide_explanation=True,
        delimiters = ("{{{", "}}}"),
    )

Aditya T.

·

llm_classify does not like it:

TypeError: got an unexpected keyword argument 'delimiters

Xander S.
·
It's actually an argument on the template parameter. So you could do something like LOGIC_LLM_JUDGE_PROMPT.delimiters = ...
Xander S.
·
Or pass the delimiters when you define that template

21 comments