Apache Airflow

Description

In this course, you will :

• Learn Apache Airflow from the ground up with proper HANDS-ON examples.
• Begin by implementing the Airflow core nomenclature - DAG, Operators, Tasks, Executors, Cfg file, UI views, and so on.
• ADVANCE Airflow concepts, which are not well explained even in Airflow's official documentation.
• XComs, Hooks, Pools, SubDAGs, Variables, Connections, Plugins, Adhoc queries, Branching, Sensors, and a plethora of other features.
• Data Profiling, Charts, Trigger rules, airflowignore file, Zombies, Undeads, LatestOnly operator are all exclusive features.
• Using Apache Airflow, create a data pipeline for Real-Time case studies.
• The resources tab contains codes and data-sets. This will save you typing time.

Syllabus :

1. Architecture Of Apache Airflow

• Architecture of Airflow
• Life Cycle of a Task

 

2. Installation

• Docker Installation (Docker Desktop)
• Docker Installation (Docker Toolbox)
• Airflow Installation

 

3. Understanding Directories in Airflow Project

• Understanding Compose files
• Understanding other directories

 

4. Airflow's UI Tour

• First look of Airflow UI
• Running default DAG in UI
• Views in UI (Tree, Graph, Gantt, etc)
• Understanding DAG definition file
• DAG file execution

 

5. What are Operators

• What are Operators
• Create a DAG

 

6. Project - Store Sales Report

• Project Requirements
• Writing Project Compose File
• Writing Project's DAG File
• Creating Connections in UI
• Writing DAG continued
• Running project DAG in UI
• 'with' - Context Manager

 

7. Airflow CLI Tour

• Running Project DAG in Airflow CLI
• Airflow CLI Commands

 

8. Executors in Airflow

• What are Executors
• Sequential Executor
• Local Executor
• Celery Executor

 

9. XComs and Variables (Advance)

• XComs in Airflow
• Implementing XComs 
• Variables in Airflow
• Dummy Operator

 

10. Advance Concepts of Airflow

• Hooks
• Pools
• Airflow Sensors
• Backfill and Catchup
• Branching
• airflow.cfg file
• LatestOnlyOperator
• Create a DAG

 

11. Data Profiling in Airflow

• Adhoc Queries
• Querying Metadata Tables
• Charts in Airflow

 

12. SubDAGs (Advance)

• What are SubDAGs
• SubDAGs implementation

 

13. PLUGINS - Creating Custom Components

• Plugins Introduction
• Create first Plugin
• Create Custom Operator
• Running Custom Operator's DAG
• Create Custom Sensor
• Create Custom Hook

 

14. Trigger Rules, airflowignore file, Zombies & Undeads

• Trigger Rules
• .airflowignore file
• Zombies
• Undeads

Show more