Efficient Data Processing on Kubernetes with PySpark, Pandas & Polars Training Course

Module 1: Big Data & Spark Foundations

• Overview of the Big Data ecosystem and Spark's role in contemporary data platforms
• Exploring Spark architecture: driver, executors, cluster manager, lazy evaluation, DAGs, and execution planning
• Distinguishing between RDD and DataFrame APIs and identifying optimal use cases for each
• Initializing and configuring SparkSession to understand fundamental application settings

Module 2: PySpark DataFrames

• Ingesting and exporting data from enterprise sources and formats: CSV, JSON, Parquet, and Delta
• Mastering PySpark DataFrames: transformations, actions, column expressions, filtering, joins, and aggregations
• Implementing advanced operations such as window functions, timestamp handling, and nested data manipulation
• Integrating data quality checks and writing reusable, maintainable PySpark code

Module 3: Kubernetes & Docker Fundamentals for Spark Users

• Conceptual understanding of how Docker images and Kubernetes pods map to Spark driver and executor processes—viewed from a cluster consumer's perspective rather than an administrator's
• Tracing the lifecycle of a Spark job on Kubernetes: pod scheduling, resource requests/limits, and mapping session settings to pod resources
• Defining the boundary between Spark session parameters controlled by the user and cluster provisioning managed by the platform team—and why this distinction is critical for tuning
• Hands-on exercise: Inspect a live Spark-on-Kubernetes job in a sandbox cluster to identify driver and executor pods

Module 4: Spark Session Configuration on Kubernetes — Deep Dive

• Analyzing executor count vs. size: navigating memory-versus-cores trade-offs and reasoning through optimal configurations
• Driver and executor memory allocation, overhead management, and translating these into Kubernetes pod resources
• Dynamic allocation: behavior on Kubernetes, scenarios where it conserves resources, and cases where it does not
• Hands-on exercise: Execute identical jobs with varying executor and core configurations to compare runtime and resource consumption

Module 5: Scaling Behaviour & Cost Optimisation

• Evaluating how node addition or removal affects job behavior, completion time, and resource usage
• Comparing many small executors against fewer large ones: analyzing performance and cost implications
• Shuffle dynamics and partition tuning, including estimating the cost impact of configuration decisions
• Hands-on exercise: Scale a sandbox cluster from five to ten nodes to observe effects on job completion time and resource consumption

Module 6: Efficient Data Ingestion & Partitioning

• The small-files problem: understanding why sources split into many 1–5 MB Parquet files degrade performance and distort partitioning
• Strategies for repartitioning and coalescing data
• Controlling partition sizes during read and write operations
• Optimizing Parquet writes to prevent reintroducing the small-files issue downstream
• Hands-on exercise: Load a dataset of small Parquet partitions, apply various repartitioning strategies, and compare performance improvements

Module 7: Pandas Memory Management & Failure Diagnosis

• Identifying root causes and symptoms of out-of-memory errors in Pandas
• Implementing memory-efficient conversion patterns between Spark and Pandas
• Preventing memory spikes when exporting large datasets to CSV
• Leveraging chunked processing and dtype optimization for memory-constrained environments
• Hands-on exercise: Recreate a typical Pandas out-of-memory scenario and resolve it using chunking and dtype optimization

Module 8: Polars as a Complementary Tool

• Positioning Polars relative to Pandas: analyzing performance characteristics, lazy evaluation, and memory behavior
• Determining where Polars fits alongside PySpark and Pandas in modern cloud data stacks and migration roadmaps, including AWS environments
• Hands-on exercise: Refactor a Pandas-heavy transformation into Polars to compare memory usage and processing speed

Module 9: Applying Optimisation to ETL & ML Workloads

• Applying configuration, partitioning, and memory management principles across a realistic ETL pipeline
• Understanding optimization considerations specific to machine learning workloads sharing the same cluster
• Implementing a systematic tuning workflow to diagnose cost and performance issues
• Hands-on exercise: Complete an end-to-end mini-project involving data loading, transformation, and simple model training, with participants independently tuning Spark configurations