Productionalizing Data Pipelines with Apache Airflow

Description

In this course, you will :

• Learn how to use Apache Airflow to master them.
• Learn about Airflow and how it generates Data Pipelines.
• Learn how to make your pipelines more resilient and predictable.
• Discover how to assign tasks using Celery and Kubernetes Executors.
• You will have the Apache Airflow skills and knowledge required to make any Data Pipelines production grade.

Syllabus :

1. Dissecting the Components of a Pipeline

• Architecture of Apache Airflow
• Installing Apache Airflow Locally
• How Do We Represent a Pipeline in Airflow?
• Our First DAG5mDissecting DAGs: Tasks and Operators
• Creating Our Pipeline 
• Creating Our Pipeline 

 

2. Demystifying Common DAGs Pitfalls

• When Is the DAG Going to Execute?
• Understanding start_date and schedule_interval
• Handling Non-default schedule_interval Cases
• Mastering Scheduling7mDemo: DAGs from the Future Past

 

3. Abstracting Functionality

• Using Macros and Airflow Templates
• Advanced DAG Flow with Branching 
• Our First Conditional Task
• Extending Functionality with Custom Operators
• Sharing Components with Airflow Plugins

 

4. Scaling Airflow

• Why Are My Tasks Sequential?3mSequential, Local and Celery Executors
• Understanding Concurrency and Parallelism with Local Executor
• Installing Celery Setup
• Distributing Tasks with Celery Executor
• A Note Regarding Airflow in Kubernetes

Show more