Get in Touch

Course Outline

Foundations of the Databricks Platform and Lakehouse

  • Overview of Databricks Lakehouse architecture and its key components
  • Methods for organizing workspaces and catalogs

Working with the Databricks Workspace and Notebooks

  • Navigating the workspace and developing using notebooks
  • Structuring code into reusable notebook modules

Apache Spark Architecture and Execution Models

  • Understanding Spark runtime architecture and its execution model
  • Concepts of lazy evaluation and the job Directed Acyclic Graph (DAG)

PySpark DataFrames and the DataFrame API

  • DataFrame abstractions and schema definitions
  • Core DataFrame operations and column expression logic

Converting SQL to PySpark DataFrames

  • Translating standard SQL clauses into DataFrame operations
  • Implementing window functions and aggregations in PySpark

Data Reading and Writing within Databricks

  • Ingesting data from common file and database sources
  • Writing and partitioning data within the Lakehouse architecture

Delta Lake and Table Management

  • Delta tables and ACID transaction support
  • Features for time travel and schema evolution

Data Cleaning and Transformation Patterns

  • Techniques for data cleaning and type conversion
  • Creating reusable transformation logic

User-Defined Functions and Modular Coding

  • Implementing Python UDFs and pandas UDFs
  • Refactoring procedural logic into modular functions

Performance Tuning and Optimization

  • Strategies for partitioning and caching
  • Identifying performance bottlenecks using the Spark UI

Essentials of Structured Streaming

  • Differences between batch and streaming processing models
  • Streaming DataFrames and basic aggregation techniques

Databricks Jobs and Workflow Orchestration

  • Scheduling notebooks as automated jobs and tasks
  • Constructing multi-step workflows with defined dependencies

Unity Catalog and Data Governance

  • Unity Catalog architecture and namespace management
  • Access control mechanisms and data lineage tracking

Testing, Debugging, and Production Best Practices

  • Unit testing PySpark logic
  • Debugging techniques and maintaining code quality standards

End-to-End Financial Services Applications

  • Developing a complete banking ETL pipeline
  • Converting legacy SQL processes to PySpark

Migrating SQL Workloads to PySpark

  • Strategies for migration planning and execution
  • Incrementally converting SQL workflows to PySpark

Requirements

  • Proficiency in Python programming, covering functions and data types
  • Familiarity with SQL concepts, such as joins, aggregations, and subqueries
  • No previous experience with Databricks or PySpark is necessary

Target Audience

  • Data engineers, data analysts, and other data professionals
  • Teams in the process of migrating existing SQL-based workflows to Databricks and PySpark
 35 Hours

Number of participants


Price per participant

Testimonials (1)

Upcoming Courses

Related Categories