Fine-tuning Multimodal Embeddings on Custom Text-Image Pairs

Save to stack
Shaw Talebi Open on YouTube ↗
Hugging Face Company Python for AI Use Case Fine-Tuning Technique Open Source LLM Model Vector Search Technique Semantic Search Topic CLIP Technique Sentence Transformers Tool Multimodal RAG Topic Contrastive Learning Technique LLM Fine-Tuning Technique Developers Audience Intermediate Level Python Tool
From the creator

🤝 Your team not maximizing Claude? I run 1:1 and team AI workshops for companies doing $10M+ per year: https://aibuilder.academy/yt/W4s6b2ZM6kI In this video, I walk through how to fine-tune CLIP on my YouTube titles and thumbnails using the Sentence Transformers Python library. Resources: 📰 Blog: https://medium.com/towards-data-science/fine-tuning-multimodal-embedding-models-bf007b1c5da5?source=friends_link&sk=df620c0d7aa4959d566771fe09766c41 💻 GitHub Repo: https://github.com/ShawhinT/YouTube-Blog/tree/main/multimodal-ai/4-ft-mm-embeddings 🤗 Model: https://huggingface.co/shawhin/clip-title-thumbnail-embeddings 💿 Dataset: https://huggingface.co/datasets/shawhin/yt-title-thumbnail-pairs References: [1] https://youtu.be/YOvxh_ma5qE [2] arXiv:2103.00020 [cs.CV] [3] arXiv:1705.00652 [cs.CL] [4] https://youtu.be/hOLBrIjRAj4 Intro - 0:00 Multimodal Embeddings - 0:44 0-shot Use Cases - 2:30 Limitations of CLIP - 3:50 Fine-tuning CLIP - 5:14 Step 1: Gather training data - 6:46 Step 2: Preprocess data - 15:20 Step 3: Define evals - 17:20 Step 4: Fine-tune model - 19:22 Step 5: Evaluate model - 26:04

Choose to Build with AI
Matched to Hugging Face

AI Maker Residence 3

The third AI workshop taught by our legendary teacher, Nick Sarafa. This is a full-day hands-on training workshop for purposeful co-creation with AI using Claude Code. Imagine having access to hundreds of billions of dollars of computing power and knowing exactly how to make it work for you through the power of super intelligence.

◆ Fri 09 Oct 2026 ◆ KOKO Cafe, London ◆ With Nick Sarafa
Get your seat See every event
AI Maker Residence 3
Live event
AI Maker Residence 3
Fri 09 Oct 2026

More like this

Running one yourself?

List your AI event,
wherever it is.

A meetup, a workshop, a hackathon, a conference. Any city, or online. Tell us about it and it lands in front of people already learning this stuff.

List your AI event