Course Outline
Databricks Platform and Lakehouse Fundamentals
- Understanding the Databricks Lakehouse architecture and its components
- Organizing workspaces and catalogs
Databricks Workspace and Notebooks
- Navigating the workspace and engaging in notebook-based development
- Structuring code into reusable notebooks
Apache Spark Architecture and Execution
- Understanding the Spark runtime architecture and execution model
- Comprehending lazy evaluation and the job DAG
PySpark DataFrames and the DataFrame API
- Exploring DataFrame abstractions and schemas
- Mastering core DataFrame operations and column expressions
Translating SQL to PySpark DataFrames
- Converting core SQL clauses into DataFrame operations
- Utilizing window functions and aggregations in PySpark
Reading and Writing Data in Databricks
- Reading from standard file and database sources
- Writing and partitioning data within the Lakehouse
Delta Lake and Table Management
- Managing Delta tables and ACID transactions
- Implementing time travel and schema evolution
Data Cleaning and Transformation Patterns
- Performing data cleaning and type conversion
- Developing reusable transformation logic
User-Defined Functions and Modular Code
- Working with Python UDFs and pandas UDFs
- Modularizing procedural logic into functions
Performance Tuning and Optimization
- Adopting partitioning and caching strategies
- Identifying bottlenecks using the Spark UI
Structured Streaming Fundamentals
- Distinguishing between batch and streaming processing models
- Utilizing Streaming DataFrames and basic aggregations
Databricks Jobs and Workflow Orchestration
- Scheduling notebooks as jobs and tasks
- Constructing multi-step workflows with dependencies
Unity Catalog and Data Governance
- Understanding Unity Catalog architecture and namespaces
- Managing access control and data lineage
Testing, Debugging, and Production Practices
- Conducting unit testing for PySpark logic
- Debugging and adhering to code quality standards
End-to-End Financial Services Use Cases
- Constructing an end-to-end banking ETL pipeline
- Translating legacy SQL processes into PySpark
Migrating SQL Workloads to PySpark
- Establishing migration strategy and planning patterns
- Executing incremental conversion of SQL workflows to PySpark
Requirements
- Proficiency in Python programming, encompassing functions and data types
- Understanding of SQL, including joins, aggregations, and subqueries
- No prior experience with Databricks or PySpark is necessary
Audience
- Data engineers, data analysts, and other data professionals
- Teams transitioning existing SQL-based workflows to Databricks and PySpark
Testimonials (1)
I liked that it was practical. Loved to apply the theoretical knowledge with practical examples.
