In this video I show you how to use Google's implementation of Sentencepiece tokenizer for question and answering systems. We will be implementing the tokenizer with offsets for albert that you can use with many different transformer based models and changing the data processing function learned from previous tutorials.
If you are not familiar with previous videos, watch these:
https://www.youtube.com/watch?v=XaQ0CBlQ4cY
https://www.youtube.com/watch?v=6a6L_9USZxg
The code implemented in this video can be found here:
https://www.kaggle.com/abhishek/sentencepiece-tokenizer-with-offsets
If you want to be on the Talk Show, fill out this form: https://bit.ly/AbhishekTalks
Follow me on:
Twitter: https://twitter.com/abhi1thakur
LinkedIn: https://www.linkedin.com/in/abhi1thakur/
Kaggle: https://kaggle.com/abhishek
Choose to Build with AI
Matched to Summarise Documents
AI Maker Residence 3
The third AI workshop taught by our legendary teacher, Nick Sarafa. This is a full-day hands-on training workshop for purposeful co-creation with AI using Claude Code.
Imagine having access to hundreds of billions of dollars of computing power and knowing exactly how to make it work for you through the power of super intelligence.
◆ Fri 09 Oct 2026◆ KOKO Cafe, London◆ With Nick Sarafa
