Course Outline
Foundations of the Databricks Platform and Lakehouse
- Overview of Databricks Lakehouse architecture and its key components
- Methods for organizing workspaces and catalogs
Working with the Databricks Workspace and Notebooks
- Navigating the workspace and developing using notebooks
- Structuring code into reusable notebook modules
Apache Spark Architecture and Execution Models
- Understanding Spark runtime architecture and its execution model
- Concepts of lazy evaluation and the job Directed Acyclic Graph (DAG)
PySpark DataFrames and the DataFrame API
- DataFrame abstractions and schema definitions
- Core DataFrame operations and column expression logic
Converting SQL to PySpark DataFrames
- Translating standard SQL clauses into DataFrame operations
- Implementing window functions and aggregations in PySpark
Data Reading and Writing within Databricks
- Ingesting data from common file and database sources
- Writing and partitioning data within the Lakehouse architecture
Delta Lake and Table Management
- Delta tables and ACID transaction support
- Features for time travel and schema evolution
Data Cleaning and Transformation Patterns
- Techniques for data cleaning and type conversion
- Creating reusable transformation logic
User-Defined Functions and Modular Coding
- Implementing Python UDFs and pandas UDFs
- Refactoring procedural logic into modular functions
Performance Tuning and Optimization
- Strategies for partitioning and caching
- Identifying performance bottlenecks using the Spark UI
Essentials of Structured Streaming
- Differences between batch and streaming processing models
- Streaming DataFrames and basic aggregation techniques
Databricks Jobs and Workflow Orchestration
- Scheduling notebooks as automated jobs and tasks
- Constructing multi-step workflows with defined dependencies
Unity Catalog and Data Governance
- Unity Catalog architecture and namespace management
- Access control mechanisms and data lineage tracking
Testing, Debugging, and Production Best Practices
- Unit testing PySpark logic
- Debugging techniques and maintaining code quality standards
End-to-End Financial Services Applications
- Developing a complete banking ETL pipeline
- Converting legacy SQL processes to PySpark
Migrating SQL Workloads to PySpark
- Strategies for migration planning and execution
- Incrementally converting SQL workflows to PySpark
Requirements
- Proficiency in Python programming, covering functions and data types
- Familiarity with SQL concepts, such as joins, aggregations, and subqueries
- No previous experience with Databricks or PySpark is necessary
Target Audience
- Data engineers, data analysts, and other data professionals
- Teams in the process of migrating existing SQL-based workflows to Databricks and PySpark
Testimonials (1)
I liked that it was practical. Loved to apply the theoretical knowledge with practical examples.