Setup datasets and experiments in Arize Updating your prompts can feel like guessing. You find a new prompting technique on arXiv or Twitter which works well on a few examples, only to later run into issues. The reality of AI engineering is that prompting is non-deterministic; it’s easy to make a small change and cause performance regressions in your product. A better approach is evaluation-driven development; leveraging Arize, you can curate a dataset of key points that you’re trying to test, run your LLM task against those key points, and use code or LLMs or user-generated annotations to evaluate the output with aggregate scores. This allows you to test as you build and verify experiments before you deploy to customers. I run through a quick demo and accompanying notebook creating a user research AI and how I iterate on the prompts in the video below! https://youtu.be/4PNU5mTbGec
Arize now runs automated actions on your data, including LLM evaluations. With just a few clicks, you can run evaluations on your traces without writing code. In this quick demo, I will show you how to setup evals in Arize on hallucination, correctness, user frustration, and more! https://youtu.be/eyvcRAjXimw
Tutorial on running evals step by step The standard for evaluating text has always been human labeling. But human evaluation is much more expensive to setup and maintain. AI engineers are now relying on LLMs to evaluate the performance of their applications. We built the open source Arize Phoenix LLM Evals library for simple, fast, and accurate LLM-based evaluations. Get a quick taste in this demo – the second in a series – exploring how you can evaluate your LLM app for things like hallucinations leveraging Phoenix. https://www.youtube.com/watch?v=5yjcbQDLLnw&feature=youtu.be
My second video for AI developers is about token counts. Yes, really. The number of tokens used by your LLM application can be really useful in uncovering patterns and issues. With token counting, you can answer questions like: are there too many tokens in my context window? Which spans are using up the most tokens in the LLM chain? Are there problematic sessions or traces that are too long or short? Which parts of an LLM application are costing the most? This tutorial dives into how Arize calculates token count and useful ways to analyze token counts with the Arize platform. Watch: https://www.youtube.com/watch?v=mpifjVPXPdE
I'm starting a video series to help AI developers create applications that are reliable, accurate, and fast! It all starts with tracing your LLM application. Tracing helps you surface problems automatically by breaking down what happens when you use a large language model. You can track down issues around bad responses, prompt templates, latency, function calls, retrieval workflows, and more. I've uploaded a tutorial which helps you auto-instrument your OpenAI application, ingest traces, and use the Arize platform with a code example. Happy building! https://www.youtube.com/watch?v=yEmP16OG3Sg
