ARC Prize Version 2 Launch Video! [Francois Chollet, Mike Knoop]
Francois Chollet and Mike Knoop join Tim Scarfe to announce ARC-AGI 2 and the ARC Prize 2025 contest. The new benchmark has been human-calibrated with 400 participants and adversarially designed so that frontier reasoning models score in the single digits, while every task remains solvable by at least two humans. The conversation covers the story behind testing OpenAI o3 on ARC v1 (where it scored 75-85%), what those results reveal about fluid intelligence versus brute-force capability, and why Chollet considers o3 the first model with genuine -- if limited -- fluid intelligence. Technical discussion ranges from chain-of-thought reasoning as natural language program synthesis, to the exponential failure modes of current reasoning systems on spatially complex tasks, to the deeper question of whether intelligence is best understood as a binary category or a spectrum of recombination depth. Chollet argues that intelligence is fundamentally about efficiency of knowledge acquisition and adaptation, not raw capability, and that ARC-AGI 2 is designed to measure exactly this distinction. SPONSOR MESSAGES: *** Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich. --- REFERENCES: Paper: [00:00:15] On the Measure of Intelligence https://arxiv.org/abs/1911.01547 [00:12:50] OpenAI o3 Performance on ARC v1 https://arcprize.org/blog/oai-o3-pub-breakthrough [00:18:30] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models https://arxiv.org/abs/2201.11903 [00:26:05] ARC Prize 2024: Technical Report https://arxiv.org/abs/2412.04604 [00:48:57] The Bitter Lesson http://www.incompleteideas.net/IncIdeas/BitterLesson.html [00:53:30] Decoding Strategies in Neural Text Generation https://www.mdpi.com/2078-2489/12/9/355/pdf Organization: [00:06:45] ARC Prize Foundation https://arcprize.org/ --- LINKS: Full Transcript: https://app.rescript.info/share/19ed0e8921636ce804d8780e6b03f67d Download PDF transcript: https://app.rescript.info/api/public/sessions/302bc045019e056a/pdf
