Automating Data Pipelines with Python & GitHub Actions [Code Walkthrough]
🤝 Your team not maximizing Claude? I run 1:1 and team AI workshops for companies doing $10M+ per year: https://aibuilder.academy/yt/wJ794jLP2Tw This is the 6th video in a series on Full Stack Data Science. Here, I use Python and GitHub actions to automate a data pipeline for FREE! More Resources: 💻 Example Code: https://github.com/ShawhinT/data-pipeline-example 📰 Read more: https://medium.com/towards-data-science/automating-data-pipelines-with-python-github-actions-c19e2ef9ca90?sk=4183bee82d107ba44750ac4bf1e1cdd1 🛠️ Data Engineering: https://youtu.be/sNa_uiqSlJo 👨🏻💻 ML app repo: https://github.com/ShawhinT/yt-search 🔍 ML app UI: https://huggingface.co/spaces/shawhin/yt-search-ui -- Homepage: https://www.shawhintalebi.com/ Intro - 0:00 Motivation - 0:32 2 Ways to Automate - 1:28 Way 1: Orchestration Tool - 2:00 Way 2: Python + Triggers - 3:38 GitHub Actions - 5:56 Example Code: Automating ETL Pipeline - 7:42 1) Create ETL Python Script - 8:33 2) Create GitHub Repo - 12:21 3) Create Workflow .yml File - 13:22 4) Add Repo Secrets - 23:50 5) Commit and Push - 25:59 Final ML App - 28:45
