End to End Data Engineering Project using Databricks Free Edition | FMCG Domain
This is an end end to data engineering project using databricks free edition that is perfect for beginners as well as advanced users. In FMCG domain, we are building an ETL pipeline for a real life industry use case where a big retail company acquires a small one and we will build the data pipeline to consolidate data from both the companies in a single lakehouse architecture. The tech stack data engineering project is: Python, SQL, Amazon S3, Medallion architecture, Spark, BI dashboard and Genie. Signup to Databricks Free Edition: https://bit.ly/4nK0NTN Databricks foundations video: https://www.youtube.com/watch?v=761SQ9Hxbic Code, dataset and other resources: https://resources.codebasics.io/pD0lLR 🚀 Ready to go from Data Analyst to AI-Enabled Data Engineer? Our live bootcamp takes you from dashboards to production pipelines: https://resources.codebasics.io/5yaUOR 🎮 Want to learn by doing? Datakata is a free, super fun game where you build data pipelines hands-on: https://resources.codebasics.io/Ahkp3O Get Certified by Codebasics with this Databricks Mini Course : https://resources.codebasics.io/dgbaAX ⭐️ Timestamps ⭐️ 00:00:00 Intro 00:00:57 Project Introduction 00:02:57 Stakeholder Discussion 00:04:22 2 Learning Paths 00:04:40 Tech Architecture 00:09:47 Data Model 00:16:42 Signup to Databricks free edition 00:17:26 Setup Catalog. Data Import 00:27:36 AWS Account Setup 00:30:48 Upload Data to S3 00:32:50 Databricks S3 Connection 00:34:34 Dimension Data Processing 01:40:23 Fact Processing - Historical Load 02:02:11 Fact Processing - Incremental Load 02:25:51 Orchestration 02:38:07 Denormalized Table and Genie 02:43:24 Dashboarding 02:54:46 Stakeholder Review Meeting 02:55:55 Conclusion Do you want to learn technology from me? Check https://codebasics.io/?utm_source=description&utm_medium=yt&utm_campaign=description&utm_id=description for my affordable video courses. Need help building software or data analytics/AI solutions? My company https://www.atliq.com/ can help. Click on the Contact button on that website. 🎥 Codebasics Hindi channel: https://www.youtube.com/channel/UCTmFBhuhMibVoSfYom1uXEg #️⃣ Social Media #️⃣ 🧑🤝🧑 Discord for Community Support: https://discord.gg/r42Kbuk 📸 Codebasics' Instagram: https://www.instagram.com/codebasicshub/ 📝 Codebasics' Linkedin : https://www.linkedin.com/company/codebasics/ ------ 📝 Dhaval's Linkedin : https://www.linkedin.com/in/dhavalsays/ 📝 Hem's Linkedin: https://www.linkedin.com/in/hemvad/ 📽️ Hem's Instagram for daily tips: https://www.instagram.com/hemvadivel/ 📸 Dhaval's Personal Instagram: https://www.instagram.com/dhavalsays/ 🔗 Patreon: https://www.patreon.com/codebasics?fan_landing=true
