What you'll learn
In this course, we will teach you how to build an end-to-end data engineering project on Azure.
You'll learn to gather data from various sources, store it in Azure Data Lake Storage, process it with Azure Data Factory, and apply analytics to the processed
This course is suitable for anyone with knowledge of SQL and Python. Whether you're a software engineer, data analyst, or IT professional
You'll gain valuable insights into building scalable and reliable data pipelines on Azure
Requirements
Basic Knowledge of Azure, Databricks
Description
In this course, we will teach you how to build an end-to-end data engineering project on Azure. You'll learn to gather data from various sources, store it in Azure Data Lake Storage, process it with Azure Data Factory, and apply analytics to the processed data using Azure Databricks. Throughout the course, you'll build a project from scratch, following the best practices and design patterns used by professional data engineers. By the end of the course, you'll have a deep understanding of the Azure data engineering stack and be able to apply your skills to real-world projects. This course is suitable for anyone with knowledge of SQL and Python. Whether you're a software engineer, data analyst, or IT professional, you'll gain valuable insights into building scalable and reliable data pipelines on Azure. Start your journey to becoming an Azure data engineering professional with this comprehensive course. Enroll in this course now and clear your Azure Interviews with 100% guaranteed. I have used below skills1. ADF2. Databricks3. ADLS Gen24. Blob Storage5. Service Principle6. IAM7. Azure Synapse8. Rest API9. Python10. SQL11. Github12. Azure DevopsBuild Your Azure Data Engineering Skills with Hands-On Experience in an End-to-End Project - Develop Real-World Solutions and Earn In-Demand Skills Today!
Overview
Section 1: Let's understand Azure Resources and How to create them
Lecture 1 Architecture
Lecture 2 Introduction of ADF
Lecture 3 Introduction of Blob Storage
Lecture 4 Introduction of AzureDataLake Gen2
Lecture 5 Introduction of Key Vault
Lecture 6 What is SPN and How to register SPN?
Lecture 7 Introduction of SQL
Lecture 8 Introduction of Azure Databricks
Lecture 9 Introduction of Azure Logic App
Lecture 10 Introduction of Azure Automation Account
Lecture 11 What is API? and How to Register API_Key ?
Section 2: Data Preparation and Uploading into the respective sources
Lecture 12 Import files,data into Azure Blob, DataLake Gen2, SQL Database
Section 3: Ingest Data from multiple source
Lecture 13 Create LinkedServices and Grant Access to Resource
Lecture 14 Create Datasets and Ingest CSV files
Lecture 15 Copy SQL tables from Server to DataLake Gen2 using parameterization
Lecture 16 Unzip files and Ingest them into Data Lake Gen2 with single pipeline
Lecture 17 Fetch Data from API using Rest API and Web Activity to fetch credentials
Lecture 18 Setup Repository and commit ADF changes to Github Repository
Lecture 19 Updates on ADF Pipeline
Lecture 20 Create Mount Point of Azure Blob Storage using Databricks
Lecture 21 Create Mount Point of Azure DataLake GEN2 and Dumping PDF files using Databricks
Lecture 22 Copy PDF files from Azure Blob Storage using parameterization to Azure Datalake
Section 4: Cleaning Process and Data Quality Check
Lecture 23 Clean the Data using AUTOLOADER in Databricks - Part 1
Lecture 24 Clean the Data using AUTOLOADER in Databricks - Part 2
Lecture 25 Clean the Data using AUTOLOADER in Databricks - Part 3
Lecture 26 Clean the Data using Batch processing in Databricks
Lecture 27 Data Quality checks on Cleansed Layer using Delta
Lecture 28 Commit Databricks Notebook using Repo feature in GitHub
Section 5: Create Dimesions and Fact tables
Lecture 29 Create Dimesions and Fact tables
Section 6: Publish the data into SQL DataBase using ADF
Lecture 30 Publish final tables into SQL Databases using Azure Databricks Delta Lake
Section 7: Setup complete ADF Pipelines
Lecture 31 Create Master Pipeline to run the complete project
Section 8: Send Data to End user using Logic App
Lecture 32 Multiple Ways to Send Data files to End Users
Lecture 33 Optimize way to send Data Files to End Users
Section 9: Run Automation Account script using Runbook
Lecture 34 Switch to SQL DW and Start/Stop the SQL DW using Automation Account
Section 10: Data Quality Layer between Mart and Publish
Lecture 35 Build Data Checks Utility to check Mart logics in Databricks
Section 11: CI-CD using Azure Devops and Github Actions
Lecture 36 Azure Data Factory Deployment
Lecture 37 Azure Blob Storage and GEN2 files Deployment
Lecture 38 Azure Databricks Deployment using Github Action
Lecture 39 Add Trigger and Global Parameters and Fix CI-CD for Azure Data Factory
Section 12: Handle corner cases in ADF
Lecture 40 How to use Filter Activity to handle Failure Runs in ADF
Section 13: Build Dashboard and Report using PowerBI
Lecture 41 Import Data and Create Report
Students who are looking to change the job
Homepage

https://www.udemy.com/course/build-end-end-azure-data-engineering-project/