Why does the Chinese Room still haunt AI?

Keith Duggar and Tim Scarfe return for the second edition of their hosts-only philosophical steakhouse, picking up right where the last episode left off -- with Keith fresh from his appearance on Liron Shapira's Doom Debates show. The conversation starts technical and stays there for a good while, working through a precise argument about why standard autoregressive LLMs are not Turing complete in the technical sense. Keith lays out the distinction between potentially infinite and actually infinite memory, explains why the algorithms learned through gradient descent on fixed-context-window architectures come from the finite state automata class rather than the Turing machine class, and confronts the practical objection head-on: even if this is technically true, does it matter? Tim pushes back with RAG systems that expand effective memory, and the two work through whether closing the read-write loop could unlock a richer algorithm space. The middle hour is a sustained engagement with John Searle's Chinese Room argument. They dig into the gap between syntax and semantics, why gears churning out responses lack something even if they pass behavioral tests, and what Searle means by requiring the right causal structure for understanding. Bishop's Dancing with Pixies reductio, Wolfram's computational boundedness, and Friston's temporal-counterfactual depth model of self-awareness all get pulled into the conversation. Keith makes one of the more interesting concessions: he cannot rule out that a sufficiently complex program with the right causal structure could cross the threshold into genuine understanding. The final act covers the Nobel Prize controversy around deep learning (Keith declines to judge but notes academia's political problems), Chomsky's critique that neural networks are not scientific theories, and a framework Keith developed on Doom Debates: focusing on AI harm rather than AI doom as a way to build broader policy coalitions. Tim distinguishes AI ethics from AI safety and both agree the speculative existential arguments, while philosophically legitimate, should not drive policy at the same level as the concrete social harms already visible. Recorded Friday 11th October 2024. --- REFERENCES: video: [00:00:30] Keith Duggar on Doom Debates https://www.youtube.com/watch?v=4v-Qh3JQ4Jc [00:00:30] Is o1 Reasoning? (MLST Philosophical Steakhouse 1) https://www.youtube.com/watch?v=nO6sDk6vO0g [00:55:00] J. Mark Bishop on MLST https://www.youtube.com/watch?v=_KVAzAzO5HU [00:55:10] Searle Google Talk https://www.youtube.com/watch?v=rHKwIYsPXLg paper: [00:10:50] On the Measure of Intelligence https://arxiv.org/abs/1911.01547 [00:31:40] Minds, Brains, and Programs https://home.csulb.edu/~cwallis/382/readings/482/searle.minds.brains.programs.bbs.1980.pdf [00:55:00] Dancing with Pixies https://philarchive.org/rec/BISDWP-2 [00:55:00] Artificial Intelligence Is Stupid and Causal Reasoning Will Not Fix It https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2020.513474/full [00:58:04] Nestedly Recursive Functions https://writings.stephenwolfram.com/2024/09/nestedly-recursive-functions/ [01:33:00] Deconstructing the AI Myth: Fallacies and Harms of Algorithmification https://www.researchgate.net/publication/382802495_Deconstructing_the_AI_Myth_Fallacies_and_Harms_of_Algorithmification [01:33:30] What Is the Philosophy of Information https://www.researchgate.net/publication/225070243_What_Is_the_Philosophy_of_Information concept: [00:31:40] Chinese Room Argument https://plato.stanford.edu/entries/chinese-room/ book: [00:58:04] Godel, Escher, Bach: An Eternal Golden Braid https://www.amazon.co.uk/Godel-Escher-Bach-Eternal-Golden/dp/0465026567 [01:22:20] Principles of Deep Learning Theory https://www.amazon.com/Principles-Deep-Learning-Theory-Science/dp/1316519333 --- LINKS: Full Transcript: https://app.rescript.info/share/e6d0ea728cf3b83bbabf3e0bdf52e036 Download PDF transcript: https://app.rescript.info/api/public/sessions/68b8550291209502/pdf