In this course, you will learn:
- How it can enhance your data science work.
- How it's the platform's answer to many big data challenges
- This practical, hands-on course helps you get comfortable with PySpark
- The Spark ecosystem, detailing its advantages over other data science platforms, APIs, and tool sets.
- Resilient Distributed Datasets (RDDs), the building blocks of Spark.
- Apache PySpark
- What you should know
1. Introduction to Apache Spark
- The Apache Spark ecosystem
- Why Spark?
- Spark origins and Databricks
- Spark components
- Partitions, transformations, lazy evaluations, and actions
2. Technical Setup
- Set up the lab environment
- Download a dataset
3. Working with the DataFrame API
- The DataFrame API
- Working with DataFrames
- Working with columns
- Working with rows
- Built-in functions
- Working with dates
- User-defined functions
- Working with joins
5. Resilient Distributed Datasets (RDDs)
- Working with RDDs