Data Engineering 101 with Kestra

Description

In this course, you will :

• Get to know Data Engineering 101 in a concise but educational manner. 
• Learn about Kestra, a cutting-edge data orchestration solution for Big Data. 
• Practicals will help you comprehend Kestra more fully. 
• Understand the best ways to use Kestra efficiently.

Syllabus:

1. Introduction to Data Engineering

• What is Data Engineering?
• Data Engineering Processes - Part 1
• Data Engineering Processes - Part 2
• Batch Processing v/s Stream Processing
• ETL v/s ELT
• Data Lake v/s Data Warehouse
• Change Data Capture

2. Introduction to Kestra

• Kestra: Orchestration Platform for Engineers
• Kestra UI Overview
• Flows and Tasks
• Namespaces

3. Kestra Architecture and Installation

• Kestra Architecture
• Installing Kestra using Docker Compose
• Other Kestra Installations

4. ETL using Kestra

• Flow Architecture
• Installing Postgres and MongoDB
• Installing Kestra with Secrets
• Flow Creation - HTTP Download
• Flow Creation - Using Postgres Plugin
• Flow Creation - Query Postgres and Upload Data to MongoDB
• Outputs and Metrics
• Plugin Defaults

5. Kestra Triggers

• Introduction to Triggers
• Triggers in Action

6. Kestra Industry Adoption

• Kestra Plugins
• Kestra Adoption