🤝 Your team not maximizing Claude? I run 1:1 and team AI workshops for companies doing $10M+ per year: https://aibuilder.academy/yt/6VcXukJuGA0
This is the 4th video in a series on AI agents. Here, I discuss how to improve LLM systems with automated feedback loops and share example code for an Upwork profile rewriter using OpenAI's Responses API.
📰 Blog: https://shawhin.medium.com/llm-in-a-loop-improving-outputs-with-evals-5620e00f7258?sk=95956863ff584b8d1fd3664b0ec8a6bc
💻 Code: https://github.com/ShawhinT/YouTube-Blog/tree/main/agents
▶️ Series Playlist: https://www.youtube.com/playlist?list=PLz-ep5RbHosU02OKABBkbsQrYWmQfoZMH
References
[1] arXiv:2402.06627 [cs.LG]
[2] https://youtu.be/-sL7QzDFW-4
Intro - 0:00
Motivation - 0:16
LLM in a Loop - 1:57
3 Types of Evals - 4:42
Type 1: Rule-based - 5:32
Type 2: LLM-based - 7:32
Type 3: Real-world - 9:00
Example: Upwork Profile Rewriter - 10:38
Limitations - 22:59
Choose to Build with AI
Matched to AI Agents
AI Maker Residence 3
The third AI workshop taught by our legendary teacher, Nick Sarafa. This is a full-day hands-on training workshop for purposeful co-creation with AI using Claude Code.
Imagine having access to hundreds of billions of dollars of computing power and knowing exactly how to make it work for you through the power of super intelligence.
◆ Fri 09 Oct 2026◆ KOKO Cafe, London◆ With Nick Sarafa
