The ARC Prize 2024 Winning Algorithm [Daniel Franzen and Jan Disselhoff]
SPONSOR MESSAGES: *** CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Check out their super fast DeepSeek R1 hosting! https://centml.ai/pricing/ Daniel Franzen and Jan Disselhoff -- the 'ARChitects' -- won the ARC Prize 2024 with co-researcher David Hartmann, achieving a remarkable 53.5% accuracy on the ARC challenge using a 12-billion parameter language model. Filmed at Tufa Labs in Zurich, they walk through their solution architecture in detail: how they tokenized grid-based visual puzzles as text and fed them directly into an LLM, why test-time training on evaluation examples gave a major score boost, and how depth-first search over token probabilities outperformed greedy and beam search for solution generation. The conversation gets into the surprising computational capabilities of language models on spatial reasoning tasks. The team found that LLMs could handle 2D grid reasoning without explicit positional encodings, that symmetry augmentations served as a powerful validation mechanism rather than a training aid, and that the model's second-best solutions were often conceptually correct -- just wrong in a specific detail like rotation direction. They also discuss why their fine-tuned 12B model outperformed much larger foundation models, the role of LoRA in preventing catastrophic forgetting during test-time training, and what the entropy distribution of their search trees reveals about how the model represents uncertainty across different task types. --- REFERENCES: Paper: [00:01:00] The ARChitects: Winning ARC Prize 2024 Solution https://github.com/da-fr/arc-prize-2024/blob/main/the_architects.pdf [00:03:38] Robustness of Analogical Reasoning in LLMs https://arxiv.org/html/2411.14215 [00:14:58] Search Methods in Language Models https://arxiv.org/html/2408.00724v2 [00:22:28] GPT-4 Code Solutions for ARC (50% SOTA) https://redwoodresearch.substack.com/p/getting-50-sota-on-arc-agi-with-gpt [00:53:08] Overcoming Catastrophic Forgetting https://www.pnas.org/doi/10.1073/pnas.1611835114 [00:53:58] LoRA: Low-Rank Adaptation of Large Language Models https://arxiv.org/abs/2106.09685 Tool: [00:07:48] Re-ARC Dataset Generator https://github.com/michaelhodel/re-arc --- LINKS: Full Transcript: https://app.rescript.info/share/57e5d773f2d0b195cbce7eee1f53aef2 Download PDF transcript: https://app.rescript.info/api/public/sessions/7772acbf1f11f44b/pdf Daniel Franzen https://github.com/da-fr REFS [00:01:05] Winning ARC 2024 solution using 12B param model, Franzen, Disselhoff, Hartmann https://github.com/da-fr/arc-prize-2024/blob/main/the_architects.pdf [00:07:50] Re-ARC dataset generator for ARC task variations, Michael Hodel https://github.com/michaelhodel/re-arc [00:22:30] GPT-4 guided code solutions for ARC tasks, Ryan Greenblatt https://redwoodresearch.substack.com/p/getting-50-sota-on-arc-agi-with-gpt
