François Chollet on OpenAI o-models and ARC

SPONSOR MESSAGES: *** CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. https://centml.ai/pricing/ Francois Chollet joins Tim Scarfe to discuss the outcomes of the 2024 ARC-AGI Prize, his departure from Google to start a new research lab focused on program synthesis, and why he believes current frontier models -- including o1 -- still cannot genuinely adapt to novelty. Chollet breaks down the two paradigms that dominated the competition: deep learning-guided program synthesis (induction) and test-time training with direct prediction (transduction). Both approaches reached roughly 55% accuracy, but the striking finding is that solutions using $10 of compute matched those using $10,000. Compute is a multiplier for ideas, not a replacement for them. The conversation goes deep into Clement Bonnet's latent program search approach, Kevin Ellis's hybrid induction-transduction strategy, and the OmniArc framework that trains a single model across multiple ARC-related tasks. Chollet explains why he sees program graphs rather than token-by-token code generation as the more promising architecture for program synthesis. On the philosophical side, Chollet distinguishes two forms of reasoning -- memorized pattern application versus genuine on-the-fly recombination of cognitive building blocks. He argues that consciousness might emerge as a self-consistency mechanism needed for iterative reasoning, and that the question 'can LLMs reason?' is less interesting than 'can they adapt to novelty?' Chollet also reveals his plans for ARC-2, discusses the logarithmic relationship between compute and accuracy that his data shows, and argues that the future of programming is democratization: anyone should be able to describe what they want automated, without writing code. --- REFERENCES: person: [00:00:00] Francois Chollet https://scholar.google.com/citations?user=VfYhf2wAAAAJ [00:36:40] Kevin Ellis - Combining Induction and Transduction https://scholar.google.com/citations?user=5YGiV0YAAAAJ [00:45:00] Clement Bonnet - Latent Program Networks https://scholar.google.com/citations?user=UQ3IbeoAAAAJ tool: [00:00:53] Keras https://keras.io/ [00:11:00] ARC-AGI Prize https://arcprize.org/ [00:11:00] ARC-AGI Dataset https://github.com/fchollet/ARC-AGI paper: [00:16:03] On the Measure of Intelligence https://arxiv.org/abs/1911.01547 [01:16:40] o3 ARC Breakthrough https://arcprize.org/blog/oai-o3-pub-breakthrough --- LINKS: Full Transcript: https://app.rescript.info/share/44fe8a0aab7235883da9cb4744848cfe Download PDF transcript: https://app.rescript.info/api/public/sessions/7b446884aa257347/pdf