Panel discussion on ARC Prize 2024 (Zurich)

Filmed at Tufa AI Labs in Zurich in early January 2025, this panel brings together Tim Scarfe with the actual winners of the 2024 ARC Prize -- Daniel Franzen and Jan Disselhoff (the "ARChitects") -- alongside IBM Research's Michael Hersche, with Tufa Labs founder Benjamin Crouzier moderating. The conversation opens with Tim explaining why o3's performance on ARC forced him to fundamentally update his views. He'd always believed solution-space prediction was impossible for ARC-style problems, assuming you'd need programs with compositional generalization. o3 changed that, though questions about dataset contamination and the sheer compute cost ($17 per task) remain. The heart of the episode is hearing directly from Daniel and Jan about how they actually won. Their approach is elegant: they tokenize ARC grids line by line, then run a depth-first search through the LLM's token probability space. Because ARC grids have far fewer valid completions than natural language, there's a tight alignment between completion probability and correctness. They exploit this by searching below a probability threshold, augmenting problems (flipping, rotating), and multiplying probabilities across perspectives to select the right answer. It's the same model doing generation and verification -- just with shifted viewpoints. The panel then gets into informed speculation about what o3 is actually doing under the hood. Is it tree of thought? A single model doing self-play search? The consensus leans toward something like STaR (the Noah Goodman Self-Taught Reasoner approach) at training time, with sophisticated tree search during inference to find optimal chain-of-thought prefixes. Daniel drops what he calls his "spiciest take": that chain-of-thought reasoning in discrete tokens is fundamentally the wrong approach. He points to Meta's Large Concept Model as more promising -- thinking in continuous concept space rather than being forced to externalize thoughts as words. Michael Hersche pushes back, noting you're still missing proper state representation either way. The discussion closes with the perennial question of whether benchmarks test the right things, what AGI even means (Chollet's "skill acquisition efficiency" definition comes up), and whether we'd recognize AGI if it showed up wearing a jagged intelligence profile. Tim notes that using ChatGPT Pro with o1 has been "genuinely unreal" -- qualitatively different from anything before. --- REFERENCES: General: [00:00:00] ARC Prize https://arcprize.org/ [00:00:00] Tufa AI Labs https://tufalabs.ai/ [00:00:59] Jan Disselhoff https://www.linkedin.com/in/jan-disselhoff-1423a2240/ [00:01:12] Daniel Franzen https://github.com/da-fr [00:01:50] Michael Hersche - IBM Research https://research.ibm.com/people/michael-hersche--1 --- LINKS: Full Transcript: https://app.rescript.info/share/c9d448cb603038533e299712808607d1 Download PDF transcript: https://app.rescript.info/api/public/sessions/f4ebf79c7b6d4d71/pdf Daniel Franzen https://github.com/da-fr