🤝 Your team not maximizing Claude? I run 1:1 and team AI workshops for companies doing $10M+ per year: https://aibuilder.academy/yt/-sL7QzDFW-4
Here, I discuss 3 types of evals and how to use them to improve LLM apps.
📰 Blog: https://medium.com/@shawhin/how-to-evaluate-and-improve-your-llm-apps-f7b08fb7493c?sk=f2fbcd3f16b958baa4734d4a39d5b237
💻 Example Code: https://github.com/ShawhinT/YouTube-Blog/tree/main/LLMs/evals
References
[1] https://youtu.be/XGJNo8TpuVA
[2] arXiv:2501.12948 [cs.CL]
[3] arXiv:2402.01383 [cs.CL]
[4] https://hamel.dev/blog/posts/llm-judge/
[5] arXiv:2203.02155 [cs.CL]
[6] https://youtu.be/SnbGD677_u0
--
Intro - 0:00
Vibe Checks - 0:27
Evals - 3:26
Type 1: Code-based - 5:58
Type 2: Human-based - 9:34
Type 3: LLM-based - 13:34
Example: Improving y2b with LLM Judge - 15:28
Choose to Build with AI
Matched to Evals
AI Maker Residence 3
The third AI workshop taught by our legendary teacher, Nick Sarafa. This is a full-day hands-on training workshop for purposeful co-creation with AI using Claude Code.
Imagine having access to hundreds of billions of dollars of computing power and knowing exactly how to make it work for you through the power of super intelligence.
◆ Fri 09 Oct 2026◆ KOKO Cafe, London◆ With Nick Sarafa
