What you'll learn
Turn messy, real-world data into actionable insights.
Gain familiarity with tools such as Deepnote, Dagster, and Metabase.
Use Deepnote as a data engineering development environment.
Generate realistic development data for analysis and visualization.
Learn data exploration and preprocessing techniques using Python and SQL.
Clean and normalize data from various sources, such as relational databases, JSON, .xls files and more.
Set up Dagster to orchestrate your data pipeline.
Integrate the processing logic into a scalable ETL pipeline with Dagster.
Deploy your pipeline to Dagster Cloud (serverless)
Optimize processing through techniques such as parallelization or streamed processing.
Create powerful data visualizations using Metabase.
Requirements
Basic Python Knowledge
Description
Do you struggle with making data-driven decisions for your business due to scattered, inconsistent, and inaccessible data? This course is the solution! Learn to build a streamlined and efficient ETL pipeline that will allow you to turn data into actionable insights.This course teaches you how to build a system that collects data from multiple sources, normalizes it, and stores it in a consistent and accessible format. You will learn how to extract data, explore and preprocess it, and ultimately visualize it to support better decision-making and optimize business processes.Forget about big data and cluster management headaches, this course is designed to get you up and running quickly with a real-time ETL pipeline. With infrastructure costs under $50 a month, you can start seeing immediate results and return on investment for your clients or company.In the first part of the course, I will walk you through the architecture and introduce you to the tools we will be using:Deepnote, as a setup-free development environmentDagster, as the pipeline orchestratorMetabase, as a low-code data visualization platformWhile the course will introduce you to the relevant features of Deepnote and Metabase, it is mostly focused on Dagster.In the next part, we will get started by generating dummy sales data of a hypothetical company using Deepnote. The code will be provided for this. Once we have the data, the course will dive into data exploration and preprocessing techniques using Python and SQL in Deepnote, including cleaning and normalizing data from various sources such as relational and JSON data, Excel sheets, and more. We will implement the processing logic in Deepnote, then commit it to a Git repository that will be shared with Dagster.In the following section, we will wrap the business logic with Dagster operations and jobs, then deploy them to Dagster Cloud (self-hosted option also available), which will allow you to manage everything from a single, unified view. In this section, you will also learn a few tricks to speed up and optimize processing, such as parallelization or streamed processing.In the final section of this course, you'll bring your preprocessed data to life with Metabase. With a few simple clicks, even non-technical individuals will be able to create stunning, powerful visualizations that unlock the full potential of your data.By the end of this course, you'll have a comprehensive understanding of the tools used and how they work together, empowering you to provide tangible benefits to your clients or company from day one, measured in thousands or tens of thousands of dollars.The choice is yours - will you seize this opportunity to deliver massive benefits to your company or clients, and claim your fair share of the rewards?
Overview
Section 1: Introduction
Lecture 1 Welcome to the World of Data Engineering
Lecture 2 The Power of Clean, Organized Data
Lecture 3 The Skills and Tools Needed to be a Successful Data Engineer
Lecture 4 An ETL pipeline for Small and Medium-Sized Businesses
Section 2: Exploring the Tools of the ETL pipeline: Deepnote, Dagster, and Metabase
Lecture 5 DeepNote
Lecture 6 Dagster
Lecture 7 Metabase
Lecture 8 Other tools
Section 3: Designing the ETL Pipeline: From Data Sources to Dashboards
Lecture 9 Building the Solution Architecture
Section 4: Setting Up Your Development Environment and Generating Dummy Data
Lecture 10 Creating a PostgreSQL Database on Google Cloud
Lecture 11 Generating Synthetic Data of a Hypothetical Client
Lecture 12 Explanation of the data generation process (optional)
Lecture 13 Verifying the Generated Data
Section 5: Getting Started with Deepnote: An Introduction to Python and SQL for Data Explor
Lecture 14 Extracting and Viewing Data in Deepnote
Lecture 15 Digging Deeper: Identifying Data Issues
Lecture 16 Digging Deeper: Coming Up with a Strategy
Lecture 17 Creating a Database Table for Storing Normalized Data
Section 6: Data Preprocessing in Deepnote: Cleaning and Normalizing Data
Lecture 18 Preprocessing Relational data: POS Transactions
Lecture 19 Preprocessing Relation data: Crypto Transactions
Lecture 20 Preprocessing JSON Data
Lecture 21 Preprocessing Excel Sheets: Loading Files from Google Drive
Lecture 22 Preprocessing Excel Sheets: Market Transactions
Lecture 23 Refactoring Business Logic: Challenge
Lecture 24 Refactoring Business Logic: Solution
Lecture 25 Unit Testing
Section 7: Setting up the ETL pipeline with Dagster
Lecture 26 Overview of Dagster Concepts
Lecture 27 Set up Local Dagster Development
Lecture 28 Extracting Data
Lecture 29 Transforming and Loading Data
Lecture 30 Partitioned Processing
Lecture 31 Job Configuration
Lecture 32 Streamed Data Processing
Lecture 33 Processing Files
Lecture 34 Creating Dagster Schedules
Lecture 35 Creating Dagster Sensors
Lecture 36 Deploying to Dagster Cloud
Section 8: Visualizing Data in Metabase
Lecture 37 Creating Visualizations from the Processed Data
Section 9: Bonus Content
Lecture 38 Bonus Lecture
Developers seeking to build scalable and efficient ETL pipelines.,Entrepreneurs looking to leverage data for business growth.,Data analysts and scientists who want to streamline their data processing workflow.,Business professionals looking to improve their data-driven decision-making abilities.,Students and recent graduates interested in a career in data engineering.,Data managers tasked with organizing and making data accessible for analysis.,Project managers looking to implement data-driven solutions for clients or company.,Individuals interested in learning cutting-edge tools and techniques in data engineering.

Homepage

https://www.udemy.com/course/transform-data-into-insights-with-dagster-and-deepnote/