One Step Closer to the Star Trek Voice AI Assistant!
Will Williams is CTO of Speechmatics in Cambridge. In this sponsored episode - he shares deep technical insights into modern speech recognition technology and system architecture. The episode covers several key technical areas: Will Williams is CTO of Speechmatics, the Cambridge-based speech recognition company. Tim has used their API for years to caption MLST episodes, and this conversation happened in their offices -- which explains the live demo at the start where an AI moderates a political debate clip in real time. The technical meat covers how Speechmatics builds production ASR systems. Their approach is hybrid: self-supervised pre-training on unlabeled audio gets them comparable accuracy to fully supervised systems like Whisper, but with roughly 100x less labeled data. Williams explains why this matters for scaling to low-resource languages where you simply don't have thousands of hours of human-transcribed speech. The architecture discussion is detailed. Their system runs multiple operating points with different latency-accuracy tradeoffs. They pad latency up to 1.8 seconds to keep the user experience consistent rather than optimizing for raw speed. Decoding uses lattices with language model integration, which lets them rescore hypotheses and handle things like proper nouns and domain-specific vocabulary without retraining the acoustic model. Diarization -- figuring out who said what -- comes up repeatedly. Williams calls it harder than ASR itself, partly because speaker embeddings get corrupted by acoustic environments and partly because cross-talk creates genuinely ambiguous boundaries. They're pushing hard on implicit source separation but the problem remains open. The conversation also covers their testing infrastructure (mirrored production traffic catches edge cases that unit tests miss), why they resist customer-specific fine-tuning (it fragments the model and makes global improvements harder), and Williams' critique of PyTorch memory management in production settings. He argues for more direct memory allocation rather than letting the framework handle it, which is a practical concern when you're serving models at scale. Featuring: Will Williams (CTO, Speechmatics) and Tim Scarfe. --- TIMESTAMPS: 00:00:00 ASR and diarization fundamentals 00:05:25 Real-time conversational AI architecture 00:09:21 Neural network streaming and multi-modal integration 00:12:49 Enterprise voice AI and real-time translation 00:20:00 Production deployment and testing infrastructure 00:29:38 Model architecture and latency-accuracy tradeoffs 00:35:40 Lattice-based decoding and language model integration 00:44:00 ASR performance metrics and real-world evaluation 00:51:30 Ethics and privacy in speech technology 01:00:50 Self-supervised learning and low-resource languages 01:11:00 Feature engineering to automated ML 01:21:00 Infrastructure scaling and PyTorch critique 01:35:00 Future of conversational AI and Ursa 2 --- REFERENCES: paper: [00:00:05] Speechmatics PDF shownotes https://www.dropbox.com/scl/fi/d94b1jcgph9o8au8shdym/Speechmatics.pdf?rlkey=bi55wvktzomzx0y5sic6jz99y&st=6qwofv8t&dl=0 [00:10:09] GFlowNets https://arxiv.org/abs/2106.04399 [01:35:00] Ursa 2 model https://www.speechmatics.com/company/articles-and-news/ursa-2-elevating-speech-recognition-across-52-languages company: [00:01:15] Speechmatics https://www.speechmatics.com/ person: [00:01:32] Will Williams https://x.com/wjwwilliams --- LINKS: Full Transcript: https://app.rescript.info/share/c6887b6d7b214f93daad1c18d70e2eb6 Download PDF transcript: https://app.rescript.info/api/public/sessions/abeef42b31287680/pdf Will Williams, CTO, Speechmatics https://x.com/wjwwilliams
