Efficient Data Processing on Kubernetes with PySpark, Pandas & Polars Training Course

Module 1: Big Data & Spark Foundations

• Introduction to the Big Data ecosystem and the role of Spark in contemporary data platforms
• Exploring Spark architecture: drivers, executors, cluster managers, lazy evaluation, DAGs, and execution planning
• Distinguishing between RDD and DataFrame APIs and identifying appropriate use cases for each
• Creating and configuring SparkSessions, along with an understanding of fundamental application settings

Module 2: PySpark DataFrames

• Ingesting and exporting data from enterprise sources and formats: CSV, JSON, Parquet, and Delta
• Utilising PySpark DataFrames: transformations, actions, column expressions, filtering, joins, and aggregations
• Executing advanced operations such as window functions, timestamp handling, and nested data management
• Implementing data quality checks and crafting reusable, maintainable PySpark code

Module 3: Kubernetes & Docker Fundamentals for Spark Users

• Conceptualising the relationship between Docker images, Kubernetes pods, and Spark driver/executor processes, tailored for cluster consumers rather than administrators
• Tracing the execution of a Spark job on Kubernetes: pod scheduling, resource requests/limits, and the mapping of session settings to these resources
• Distinguishing between Spark session parameters under user control and cluster provisioning managed by platform teams, and why this boundary is critical for tuning
• Hands-on exercise: Examine a live Spark-on-Kubernetes job in the sandbox cluster to identify driver and executor pods

Module 4: Spark Session Configuration on Kubernetes — Deep Dive

• Analyzing executor count versus executor size: balancing memory against cores and how to evaluate these trade-offs
• Allocating memory for drivers and executors, accounting for overhead, and mapping these to pod resources
• Dynamic allocation: its behaviour on Kubernetes, scenarios where it conserves resources, and those where it does not
• Hands-on exercise: Execute identical jobs under varying executor and core configurations to compare runtime and resource consumption

Module 5: Scaling Behaviour & Cost Optimisation

• Evaluating how adding or removing nodes affects job dynamics, completion time, and resource use
• Weighing the performance and cost implications of using many small executors versus fewer large ones
• Managing shuffle behaviour and tuning shuffle partitions, including assessing the cost impact of configuration choices
• Hands-on exercise: Expand the sandbox cluster from five to ten nodes and monitor the resulting changes in job completion time and resource consumption

Module 6: Efficient Data Ingestion & Partitioning

• Analyzing the small-files problem: why sources split into numerous 1–5 MB Parquet files harm performance and skew partitioning
• Strategies for repartitioning and coalescing data
• Regulating partition sizes during both read and write operations
• Optimising Parquet writes to prevent the recurrence of small-file issues downstream
• Hands-on exercise: Load a dataset comprising many small Parquet partitions, apply various repartitioning methods, and assess performance improvements

Module 7: Pandas Memory Management & Failure Diagnosis

• Identifying the root causes of out-of-memory errors in Pandas and recognising their indicators
• Utilising memory-efficient patterns for data conversion between Spark and Pandas
• Preventing memory surges when exporting large datasets to CSV
• Leveraging chunked processing and dtype optimisation in resource-constrained environments
• Hands-on exercise: Replicate a common Pandas out-of-memory scenario and resolve it through chunking and dtype optimisation

Module 8: Polars as a Complementary Tool

• Positioning Polars in relation to Pandas: performance traits, lazy evaluation, and memory dynamics
• Identifying where Polars complements PySpark and Pandas in modern cloud data stacks and migration strategies, including AWS environments
• Hands-on exercise: Rebuild a Pandas-heavy transformation using Polars and compare memory usage and processing speed

Module 9: Applying Optimisation to ETL & ML Workloads

• Integrating configuration, partitioning, and memory management principles into a realistic ETL pipeline
• Considering optimisation factors specific to machine learning workloads running on the same cluster
• Implementing a practical tuning workflow to systematically diagnose cost and performance issues
• Hands-on exercise: Finish an end-to-end mini-project involving data loading, transformation, and basic model training, with participants adjusting Spark configurations independently