Rasmus B.

Anthony P. can splits be constructed from python?

Anthony P. oh nice, that looks perfect! 🙂

RunLLM I would still like to track the experiment with the UI/server. Can I prefilter the dataset and pass it to the experiment method, or do I need to reupload it? My use-case is something like grading the examples as either "easy" or "hard", and then being able to run either of them alone or both together.

Roger Y. looked around a bit for what others were doing, and there isn't a lot of support for modular prompts around to be fair, but I did notice that Langfuse has some functionality for it: https://langfuse.com/docs/prompt-management/features/composability

Roger Y. I'm honestly still figuring it out, but we are taking a modular approach to prompts (so prompts are composed of multiple versioned sub-prompts) and in addition to pure instructional prompts we have context prompts, e.g. we have a description of our core data model to help with using query tools. This data model then gets embedded into multiple system prompts to provide context for different operations in the agent graph. So since I also need to do post-hoc composition, in some sense it would be easier for me if I could just version pure strings/templates, whereas now I have to do string -> system message-> upload -> phoenix store -> download -> versioned prompt -> openai message -> text -> compose. To make better use of the message storage format I could maybe version the composed prompts separately, in which case I could plug it directly into Langgraph (by converting to the format), but there is some overhead with tracking how the composed prompt is pieced together from particular sub-prompt versions. Don't know if modular prompts are on the roadmap in some way or completely out of scope 🙂

Dustin N. Roger Y. I can totally see what you are going for, but feels a bit strict to require attachment of specific metadata to the prompt even if the user doesn't need it? If extended to work with frameworks like Langchain, is the idea that you'd set model_provider="LANGCHAIN" or something like that? For what it's worth, I found today that langchain_core.messages.utils.convert_to_messages converts OpenAI message formats pretty painlessly, but it still seems like an odd roundtrip to first convert it to OpenAI format.

RunLLM feel like I searched the python API and did not see the delete method. Is there otherwise a shortcut for sending REST calls through the client with the correct headers etc?

thanks 🔒[private user] feel like I searched the python API and did not see the delete method, will check again. Otherwise I will try REST. Is there a shortcut for sending REST calls through the client with the correct headers etc?