Lab - author, schedule and monitor data pipelines through practical examples using Apache Airflow and Practice Exams
What you'll learn
Develop plugins to extend the capabilities of Apache Airflow.
Master in core functionalities such as DAGs, Operators, Tasks, Workflows, etc
The difference between Sequential, Local and Celery Executors, how do they work and how can you use them.
Install and configure Apache Airflow
Understand and apply advanced concepts of Apache Airflow such as XCOMs, Branching and SubDAGs.
Incorporate Apache Airflow within a Big Data ecosystem.
Engage in critical thinking, devise solutions, and implement them using Airflow to tackle real data processing challenges.
Requirements
A very basic knowledge of Python will be an add-on.
Description
Apache Airflow is a powerful open-source platform designed for programmatically authoring, scheduling, and monitoring workflows. If you find yourself dealing with multiple ETL(s) to manage, Airflow becomes an essential tool in your arsenal.This comprehensive course will equip you with everything you need to begin harnessing the capabilities of Apache Airflow. Through a combination of theory and practical videos, you will embark on a journey starting with the fundamentals, understanding what Airflow is and how it operates. From there, we'll delve into more advanced topics, such as creating plugins and building dynamic pipelines to tackle real-world challenges effectively.What's included in the course ?Complete Apache Airflow concepts explained from Scratch to ADVANCE implementation.Each and every Airflow concept is explained with HANDS-ON examples.Includes each and every, even thin detail of Airflow.After completing this course, you can start working on any Airflow project with full confidence.Who this course is for:Those with a natural curiosity for data engineering.Individuals seeking to grasp both basic and advanced concepts of Apache Airflow.Enthusiasts who prefer a hands-on learning experience.Add-OnsQuestions and Queries will be answered very quickly.The course provides attached Airflow codes and datasets used in lectures for your convenience.I am going to update it frequently, every time adding new components of Airflow.
Overview
Section 1: Introduction
Lecture 1 What is Airflow? What is Workflow?
Lecture 2 Why Airflow?
Section 2: Install Apache Airflow on Windows
Lecture 3 Enable WSL
Lecture 4 Notes: Install WSL
Lecture 5 Install WSL
Lecture 6 Notes: Environment Setup
Lecture 7 Environment Setup
Lecture 8 Install Editor
Lecture 9 Notes: Install Airflow
Lecture 10 Install Airflow
Section 3: Kick start with Airflow Lab
Lecture 11 Basic Workflow with Dummy Operators in Apache Airflow
Lecture 12 Basic Workflow using BashOperator in Apache Airflow
Lecture 13 Basic Workflow using PythonOperator vs BashOperator
Section 4: Catch-up & Backfill
Lecture 14 Airflow DAG with Catchup Enabled
Lecture 15 Airflow DAG with Catchup Disabled
Lecture 16 Backfilling in Apache Airflow
Section 5: Scheduling a DAG
Lecture 17 Scheduling a DAG to Run Daily at 11:45 PM in Airflow.
Individuals with a curiosity for data engineering.,Individuals seeking to acquire knowledge of both fundamental and advanced concepts about Apache Airflow.
Screenshots