Fine-tuning Multimodal Embeddings on Custom Text-Image Pairs

Save to stack
Shaw Talebi Open on YouTube ↗
Hugging Face Company Python for AI Use Case Fine-Tuning Technique Open Source LLM Model Vector Search Technique Semantic Search Topic CLIP Technique Sentence Transformers Tool Multimodal RAG Topic Contrastive Learning Technique LLM Fine-Tuning Technique Developers Audience Intermediate Level Python Tool
From the creator

🤝 Your team not maximizing Claude? I run 1:1 and team AI workshops for companies doing $10M+ per year: https://aibuilder.academy/yt/W4s6b2ZM6kI In this video, I walk through how to fine-tune CLIP on my YouTube titles and thumbnails using the Sentence Transformers Python library. Resources: 📰 Blog: https://medium.com/towards-data-science/fine-tuning-multimodal-embedding-models-bf007b1c5da5?source=friends_link&sk=df620c0d7aa4959d566771fe09766c41 💻 GitHub Repo: https://github.com/ShawhinT/YouTube-Blog/tree/main/multimodal-ai/4-ft-mm-embeddings 🤗 Model: https://huggingface.co/shawhin/clip-title-thumbnail-embeddings 💿 Dataset: https://huggingface.co/datasets/shawhin/yt-title-thumbnail-pairs References: [1] https://youtu.be/YOvxh_ma5qE [2] arXiv:2103.00020 [cs.CV] [3] arXiv:1705.00652 [cs.CL] [4] https://youtu.be/hOLBrIjRAj4 Intro - 0:00 Multimodal Embeddings - 0:44 0-shot Use Cases - 2:30 Limitations of CLIP - 3:50 Fine-tuning CLIP - 5:14 Step 1: Gather training data - 6:46 Step 2: Preprocess data - 15:20 Step 3: Define evals - 17:20 Step 4: Fine-tune model - 19:22 Step 5: Evaluate model - 26:04

Stop watching · start building
Matched to Hugging Face

Makers residency 4th edition

Your idea can be anything. At our last workshops, people built plant-based nutrition businesses, a way to measure planetary resilience, crochet headwear and a way to connect lonely people living in the same building. An asset manager went on to raise an additional £25M on their fund within nine months. The fourth workshop is a full day at KOKO with Nick Sarafa, learning to build with Claude Code. Bring the thing you keep meaning to start. We’ll bring the pizza.

◆ Fri 13 Nov 2026 ◆ KOKO, London
Get tickets See every event
Makers residency 4th edition
Live event
Makers residency 4th edition
Fri 13 Nov 2026
Find a job · 1 live role

Jobs in AI.
Apply now.

Apply once › get screened › meet the company

See all roles

More like this

Running one yourself?

List your AI event,
wherever it is.

A meetup, a workshop, a hackathon, a conference. Any city, or online. Tell us about it and it lands in front of people already learning this stuff.

List your AI event