Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 35 hours
Course Outline
Introduction, Goals, and Migration Strategy
- Course objectives, alignment with participant profiles, and success metrics
- Strategic migration approaches and risk assessment
- Configuration of workspaces, repositories, and lab datasets
Day 1 — Migration Foundations and Architecture
- Core Lakehouse concepts, Delta Lake introduction, and Databricks architecture
- Differences between SMP and MPP and their impact on migration
- Medallion (Bronze→Silver→Gold) design principles and Unity Catalog overview
Day 1 Lab — Converting Stored Procedures
- Practical migration of a sample stored procedure to a notebook
- Translating temp tables and cursors into DataFrame transformations
- Verifying and comparing results against the original output
Day 2 — Advanced Delta Lake & Incremental Loading
- ACID transactions, commit logs, versioning, and time travel features
- Auto Loader, MERGE INTO patterns, upserts, and schema evolution
- OPTIMIZE, VACUUM, Z-ORDER, partitioning, and storage optimization
Day 2 Lab — Incremental Ingestion & Optimization
- Building Auto Loader ingestion and MERGE workflows
- Applying OPTIMIZE, Z-ORDER, and VACUUM; verifying outcomes
- Evaluating improvements in read/write performance
Day 3 — SQL in Databricks, Performance & Debugging
- Analytical SQL capabilities: window functions, higher-order functions, JSON/array management
- Interpreting Spark UI, DAGs, shuffles, stages, tasks, and identifying bottlenecks
- Query optimization strategies: broadcast joins, hints, caching, and reducing spills
Day 3 Lab — SQL Refactoring & Performance Tuning
- Refactoring complex SQL processes into optimized Spark SQL
- Using Spark UI traces to diagnose and resolve skew and shuffle problems
- Benchmarking pre- and post-tuning performance and documenting adjustments
Day 4 — Tactical PySpark: Replacing Procedural Logic
- Spark execution model: drivers, executors, lazy evaluation, and partitioning strategies
- Converting loops and cursors into vectorized DataFrame operations
- Modularization, UDFs/pandas UDFs, widgets, and creating reusable libraries
Day 4 Lab — Refactoring Procedural Scripts
- Converting procedural ETL scripts into modular PySpark notebooks
- Incorporating parametrization, unit-style tests, and reusable functions
- Conducting code reviews and applying best-practice checklists
Day 5 — Orchestration, End-to-End Pipeline & Best Practices
- Databricks Workflows: job design, task dependencies, triggers, and error management
- Designing incremental Medallion pipelines with quality rules and schema validation
- Integrating with Git (GitHub/Azure DevOps), CI, and testing strategies for PySpark logic
Day 5 Lab — Constructing a Complete End-to-End Pipeline
- Assembling a Bronze→Silver→Gold pipeline orchestrated via Workflows
- Implementing logging, auditing, retries, and automated validations
- Executing the full pipeline, verifying outputs, and preparing deployment documentation
Operationalization, Governance, and Production Readiness
- Unity Catalog governance, lineage tracking, and access control best practices
- Cost management, cluster sizing, autoscaling, and job concurrency patterns
- Deployment checklists, rollback strategies, and runbook development
Final Review, Knowledge Transfer, and Next Steps
- Participant presentations on migration work and key takeaways
- Gap analysis, recommended follow-up activities, and handover of training materials
- References, advanced learning paths, and support options
Requirements
- A solid grasp of data engineering principles
- Proficiency in SQL and stored procedures (Synapse / SQL Server)
- Knowledge of ETL orchestration concepts (ADF or similar tools)
Target Audience
- Technology managers with a background in data engineering
- Data engineers transitioning from procedural OLAP logic to Lakehouse patterns
- Platform engineers overseeing Databricks adoption
