Sarah W.

For our paper read this week we're doing something a little special (that we're working to make more regular): two of the authors will join us to talk through the paper and take your questions. The question they tackle in this one: What if LLMs could generate their own finetuning data and optimization instructions in response to new tasks or information? 🤔 ✔️ Self-Adapting LLMs (SEAL) is a framework they developed that shows a promising step towards that. It enables LLMs to generate their own fine tuning data and optimization strategies to persistently adapt their weights. Your hosts Dylan C. and Parth S. 💁‍♂️💁‍♂️ will dig into how this approach challenges traditional fine-tuning pipelines. Register here: https://lu.ma/szk3zrbd

👋 This morning's paper read will be served Extra Spicy by Dylan C. and Parth S.. 🔥 A new paper from researchers at Apple challenges today’s evaluation methods and introduces a new benchmark: synthetic puzzles with controllable complexity and clean logic. Their findings? LRMs show surprising failure modes, including a complete collapse on high-complexity tasks and a decline in reasoning effort as problems get harder. BUT THEN someone at Anthropic published a response aptly titled The Illusion of the Illusion of Thinking, which argues that their findings primarily reflect experimental design limitations rather than fundamental reasoning failures. Here's a direct link to join at 10:00am PT 👉 https://arize.zoom.us/j/89593430181

( FYI Observe ticket giveaway happening over here: https://arize-ai.slack.com/archives/C032BFSMFPH/p1749562714005479)

📣 If you missed the community rate but still want to go to Observe…we’re doing a ticket giveaway on here (and also on LinkedIn). Here’s how to enter: 💬 Reply to this message: Which speaker or topic are you most excited about, and how do you anticipate that session or topic directly impacting your work, research, or something you’re building? I’ll reach out to the winners with a code you can use. (Greater chance of winning here than on LinkedIn!! 🙂) Find the agenda and speakers 👉 arize.com/observe-2025

Recent Product Updates Roundup 🤠 Here’s everything we shipped in May: ➕ Realtime Trace Ingestion now available across all Arize AX instances ➕ Expanded OpenAI model support in Prompt Playground + Tasks ➕ Cleaner span view with improved input/output display ➕ Trace-level attribute search to find what you need faster ➕ Latency + token counts added to Prompt Playground for deeper prompt insights 👉 https://arize.com/blog/new-in-arize-realtime-trace-ingestion-prompt-playground-upgrades-more/

📅 The agenda for Observe went live last night. 📅 We also added a few new speakers in the last couple weeks--the newest one we’re excited about is Todd Underwood, Head of Reliability at Anthropic. + OpenAI, NVIDIA, Microsoft, CrewAI, Priceline, Pepsico, Figma, Elastic, Letta and many others We Also have a few researchers who have published great papers this year, including Amir Gholami (UC Berkeley) whose Plan-and-Act paper has really been making the rounds. Also excited to have folks from Stanford & UCSC. Your ticket includes lunch, happy hour, and a full day of sessions (and some really good swag). Agenda here: https://arize.com/observe-2025/agenda/ Join us June 25 in SF: https://arize.com/observe-2025/#tickets 👀 Connect with other attendees in Arize Observe and stay tuned for more updates as we get closer to.

🚨Grab your community rate for Observe before it's too late! Today is the last day to get Observe tickets for $50. After today, prices will go up. 📈 Use the code at checkout: BUILTWITHARIZE This is a Great Deal! $50 for a full day of talks, networking, community demos, lunch & happy hour. While you're at it, check out the speakers we've added so far: arize.com/observe-2025

Get an inside look at one team’s quest to transition an AI-powered travel assistant from text to voice. 🧙 Explore the challenges and opportunities of integrating audio into AI systems, as well as best practices for evaluating multimodal agents. Learn how Priceline approaches performance measurement and what the future holds for agent-based and multimodal evaluations. 📅 April 10, 10am PT Join us here 🪄 https://arize.com/resource/agents-in-the-wild-priceline

Would love insights from this community for our agents survey. Even if you aren't actively evaluating agents today, your experience will help us build a clearer picture of the landscape. This will be part of a 2025 report on agent evaluation. 🫥 It's totally anonymous unless you want the chance to win premium Arize swag and other prizes--then you just have to give us your email. 🙂 Survey: https://arize.com/ai-agent-evaluation-and-observability-survey