The Most Important Skill for AI Engineers
Thanks to Progress for sponsoring the video. checkout it out here: https://www.progress.com/agentic-rag/book-a-demo?utm_medium=paid-social&utm_source=promptengineering&utm_campaign=influencer AI evals are how you know whether an AI agent actually works. In this video, you'll learn what evals are, why testing an AI agent is different from testing regular software, and how to evaluate an agent with code checks, an LLM judge, and human review. Then we build an eval set for a text-to-SQL agent, use error analysis to find what's failing, calibrate an LLM judge against human labels, and fix the agent and rerun the evals. Further reading: Demystifying evals for AI agents (Anthropic): https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents AI Evals FAQ (Hamel Husain and Shreya Shankar): https://hamel.dev/blog/posts/evals-faq/ Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards: https://arxiv.org/abs/2601.08778 Chapters: 0:00 What are AI evals and why you need them 1:47 Why testing an AI agent is different 3:32 Model evals vs product evals 5:41 Three types of graders: code, LLM judge, human 7:50 Sponsor: Progress Agentic RAG 8:57 Hands-on: evaluating a text-to-SQL agent 10:46 When the grader is wrong 11:40 Error analysis: finding what actually fails 13:48 How to build an LLM judge 15:40 Fix the agent and rerun the evals 16:50 Limitations of this approach 17:51 How to know your AI agent works Website: https://engineerprompt.ai/ RAG Beyond Basics Course: https://prompt-s-site.thinkific.com/courses/rag Let's Connect: 🦾 Discord: https://discord.com/invite/t4eYQRUcXB ☕ Buy me a Coffee: https://ko-fi.com/promptengineering |🔴 Patreon: https://www.patreon.com/PromptEngineering 💼Consulting: https://calendly.com/engineerprompt/consulting-call 📧 Business Contact: engineerprompt@gmail.com Become Member: http://tinyurl.com/y5h28s6h 💻 Pre-configured localGPT VM: https://bit.ly/localGPT (use Code: PromptEngineering for 50% off). Signup for Newsletter, localgpt: https://tally.so/r/3y9bb0
