Efficient Data Processing on Kubernetes with PySpark, Pandas & Polars Training Course

Module 1: Big Data & Spark Fundamentals

• Overview of the Big Data landscape and Spark's role in modern data platforms
• Comprehending Spark architecture: driver, executors, cluster manager, lazy evaluation, DAG and execution planning
• Distinguishing between RDD and DataFrame APIs and determining the appropriate use case for each
• Setting up and configuring SparkSession to grasp the fundamentals of application configuration

Module 2: PySpark DataFrames

• Ingesting and exporting data from enterprise sources and formats: CSV, JSON, Parquet and Delta
• Manipulating PySpark DataFrames: transformations, actions, column expressions, filtering, joins and aggregations
• Executing advanced operations such as window functions, timestamp handling and managing nested data
• Implementing data quality checks and authoring reusable, maintainable PySpark code

Module 3: Kubernetes & Docker Essentials for Spark Users

• Understanding the relationship between Docker images and Kubernetes pods in relation to Spark driver and executor processes — a conceptual view for cluster consumers rather than administrators
• Insights into Spark job execution on Kubernetes: pod scheduling, resource requests and limits, and how session settings map to them
• Differentiating between what is controlled via Spark session parameters and what is managed by the platform team through cluster provisioning — and why this distinction is critical for tuning
• Practical Exercise: Examine an active Spark-on-Kubernetes job in the sandbox cluster to identify driver and executor pods

Module 4: Spark Session Configuration on Kubernetes — In Depth

• Analysing executor count versus executor size: memory versus cores trade-offs and how to evaluate them
• Driver and executor memory allocation, overhead considerations, and their translation into pod resources
• Dynamic allocation: behaviour on Kubernetes, scenarios where it conserves resources and those where it does not
• Practical Exercise: Execute the same job with varying executor and core configurations to compare runtime and resource consumption

Module 5: Scaling Dynamics & Cost Refinement

• Understanding how adding or removing nodes alters job behaviour, completion time and resource usage
• Evaluating the trade-offs between many small executors and fewer large ones in terms of performance and cost
• Shuffle dynamics and tuning shuffle partitions, including assessing the cost impact of configuration decisions
• Practical Exercise: Scale the sandbox cluster from five to ten nodes to observe the impact on job completion time and resource consumption

Module 6: Efficient Data Ingestion & Partitioning

• Understanding the small-files problem: why sources split across numerous 1–5 MB Parquet files degrade performance and skew partitioning
• Strategies for repartitioning and coalescing
• Managing partition sizes during read and write operations
• Writing Parquet data efficiently to prevent the recreation of small-file issues downstream
• Practical Exercise: Load a dataset comprising many small Parquet partitions, apply various repartitioning strategies and compare performance before and after optimisation

Module 7: Pandas Memory Control & Failure Analysis

• Identifying the root causes of out-of-memory errors in Pandas and recognising their symptoms
• Applying memory-efficient conversion patterns between Spark and Pandas
• Preventing memory blow-ups when exporting large datasets to CSV
• Utilising chunked processing and dtype optimisation for constrained environments
• Practical Exercise: Reproduce a standard Pandas out-of-memory scenario and resolve it using chunking and dtype optimisation

Module 8: Polars as a Complementary Tool

• Positioning Polars relative to Pandas: performance traits, lazy evaluation and memory behaviour
• Understanding where Polars fits alongside PySpark and Pandas in modern cloud data stacks and migration roadmaps, including AWS environments
• Practical Exercise: Rewrite a Pandas-centric transformation in Polars to compare memory usage and processing speed

Module 9: Applying Optimisation to ETL & ML Workloads

• Applying configuration, partitioning and memory management principles across a realistic ETL pipeline
• Understanding optimisation considerations specific to machine learning workloads running on the same cluster
• Applying a practical tuning workflow to systematically diagnose cost and performance issues
• Practical Exercise: Complete an end-to-end mini-project combining data loading, transformation and a simple model training step, with participants tuning the Spark configuration themselves