End to End Data Engineering Project using Databricks Free Edition | Spark Declarative Pipelines
In this Databricks tutorial, you will learn Lakeflow Spark Declarative Pipelines and build a real end-to-end project in the transportation domain that you can confidently put on your resume. This Databricks micro Course explains everything in a simple, intuitive way! Signup to Databricks Free Edition: https://bit.ly/4nK0NTN Code, Dataset, other resources: https://resources.codebasics.io/AvKXOB The Certification link: https://resources.codebasics.io/RuTWFY ⭐️ Timestamps ⭐️ 00:00:00 Intro 00:01:20 Project Introduction 00:03:30 Stakeholder Discussion 00:05:32 Technical Architecture & Data Understanding 00:08:25 Databricks Free Edition 00:09:10 Create Catalog and Schemas 00:11:41 S3 Bucket Setup Connecting with Databricks 00:16:41 Declarative vs Imperative Programming 00:18:19 City Table: Bronze Layer 00:31:20 City Table: Silver Layer 00:35:53 SDP Key Concepts 00:43:25 Calendar Table: Silver Layer 00:51:03 Trips Table: Bronze Layer 01:10:53 Generate BI Ready Data in Gold Layer 01:19:46 Incremental Load 01:25:20 Access Management 01:29:58 LDSP Benefits 01:32:42 Stakeholder Overview 01:34:35 What's Next Do you want to learn technology from me? Check https://codebasics.io/?utm_source=description&utm_medium=yt&utm_campaign=description&utm_id=description for my affordable video courses. Need help building software or data analytics/AI solutions? My company https://www.atliq.com/ can help. Click on the Contact button on that website. 🎥 Codebasics Hindi channel: https://www.youtube.com/channel/UCTmFBhuhMibVoSfYom1uXEg #️⃣ Social Media #️⃣ 🧑🤝🧑 Discord for Community Support: https://discord.gg/r42Kbuk 📸 Codebasics' Instagram: https://www.instagram.com/codebasicshub/ 📝 Codebasics' Linkedin : https://www.linkedin.com/company/codebasics/ ------ 📝 Dhaval's Linkedin : https://www.linkedin.com/in/dhavalsays/ 📝 Hem's Linkedin: https://www.linkedin.com/in/hemvad/ 📽️ Hem's Instagram for daily tips: https://www.instagram.com/hemvadivel/ 📸 Dhaval's Personal Instagram: https://www.instagram.com/dhavalsays/ 🔗 Patreon: https://www.patreon.com/codebasics?fan_landing=true
