The Weird ChatGPT Hack That Leaked Training Data [Dr. Yannic Kilcher / Prof. Florian Tramer]

Dr. Yannic 'lightspeed' Kilcher sits down with an AI Security Researcher Prof. Florian Tramer (an expert in the field, collaborating with folks like Nicholas Carlini and Milad Nasr at Google) to ruminate on the murky state of affairs in AI security. Out of the gate they assert that AI detectors (those tools that try to spot if text is human-written or AI-generated) are basically doomed to fail—like trying to catch an eel with greasy hands blindfolded. Lightspeed Kilcher shares a funny story about how the word "Delve" kept rearing its ugly head in AI text because of training data emanating from Nigerian crowd workers, leading to false flags in job applications. Tramer then takes the gloves off about security risks, saying that even a 99% success rate in AI means total failure in security terms—attackers will always find that 1% weak spot. They share an interesting example of a "weird attack" on ChatGPT: asking it to repeat "poem" forever made it spit out memorized internet scraps, leaking training data like a sieve. OpenAI patched it, but it's like slapping a band-aid on a shark bite. Lightspeed Kilcher asks about top worries, and Tramer lists three big ones in relatable terms: 1) Training AI on private data (like medical records) could lead to massive leaks, since models memorize stuff without us fully understanding how; 2) "Prompt injections," where bad actors hijack AI agents to do harmful things, like Anthropic's demo that controls computers—it's like giving your keys to a stranger; and 3) The rush to deploy AI without safeguards, echoing past tech blunders like SQL hacks. They reflect on how ChatGPT has lit a fire under AI security research—making it "amazing and scary" by turning hypotheticals into real-world issues with millions of users. The conversation touches on why just "scaling up" AI won't fix everything (we need causal understanding, not just more data), and why watermarking AI outputs isn't a silver bullet—it's easy to bypass, especially with open-source models. It's about time we had our old comrade light speed Kilcher back on MLST! TRANSCRIPT: https://app.rescript.info/public/share/PGieXFZCRyJjJswicatmNFia_pKCt_WVP-pZS4qguiw LINKS: Scalable Extraction of Training Data from (Production) Language Models [Poem Attack] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, Colin Raffel https://arxiv.org/abs/2311.17035 - Prof. Florian Tramer https://www.floriantramer.com/ - Yannic Kilcher's YouTube channel https://www.youtube.com/c/YannicKilcher - Nicholas Carlini's research profile (collaborator on the data recovery project): https://nicholas.carlini.com/ – Includes papers on AI attacks and memorization risks.