Capability Development / Data Engineering
Data Engineering with PySpark & Databricks
This program builds scalable data engineering skills with PySpark and the Databricks Lakehouse Platform. Participants process large datasets with DataFrames, RDDs and Spark SQL, build batch and streaming ETL pipelines, optimise Spark jobs and apply MLlib — then put it all to work on Databricks with notebooks, compute clusters, Delta Lake, data ingestion, Bronze/Silver/Gold transformations, Databricks SQL dashboards, jobs, workflows and Unity Catalog governance.
Program Overview
This program builds scalable data engineering skills with PySpark and the Databricks Lakehouse Platform. Participants process large datasets with DataFrames, RDDs and Spark SQL, build batch and streaming ETL pipelines, optimise Spark jobs and apply MLlib — then put it all to work on Databricks with notebooks, compute clusters, Delta Lake, data ingestion, Bronze/Silver/Gold transformations, Databricks SQL dashboards, jobs, workflows and Unity Catalog governance.
Audience Profile
Data engineers building distributed data pipelines; developers moving into big data processing; data analysts scaling up to large datasets; and professionals preparing to work on Spark-based platforms such as Databricks, Microsoft Fabric or Synapse.
Course Syllabus
Introduction to PySpark
Topics
- Understand Spark and PySpark
- Apply distributed computing concepts
- Create SparkSession and SparkContext
- Execute jobs on clusters
- Set up PySpark environment
Working with DataFrames and RDDs
Topics
- Work with RDD fundamentals: create RDDs, transform RDDs, use map/filter, use reduce/collect
- Use DataFrame APIs: load files, load tables, select/filter, rename/derive
- Manage schemas and data types: infer schemas, define schemas, nested fields
- Apply transformations and actions: narrow transforms, wide transforms, trigger actions, lazy execution
- Query data with Spark SQL: create temporary views, run SQL against DataFrames
Data Processing with PySpark
Topics
- Read large-scale data formats: CSV/JSON, Parquet/ORC, Avro, schema options
- Transform structured data: filter/sort, select/rename, derived columns
- Join and aggregate datasets: inner/outer joins, GroupBy, rollups
- Handle data quality issues: null handling, duplicate checks, cleansing rules
- Build advanced transformations: windows, ranking, nested data
PySpark SQL and Advanced Functions
Topics
- Apply built-in PySpark functions: string funcs, date funcs, numeric funcs, when/lit/col/expr
- Work with user-defined functions: create UDFs for custom logic, use Pandas UDFs for performance
- Execute Spark SQL queries: register tables, create views, SQL + DataFrames
PySpark for ETL
Topics
- Ingest data from multiple sources
- Cleanse and standardize datasets
- Design scalable pipelines
- Write data to target systems
- Build batch ETL workflows
Structured Streaming with PySpark
Topics
- Process streaming data: read streams, source setup, write sinks
- Manage streaming operations: triggers, checkpoints, late data, watermarking
- Build real-time pipelines: micro-batches, query status
Performance Optimization in PySpark
Topics
- Partitioning and repartitioning
- Caching and persistence
- Shuffle optimization
- Broadcast joins
- Spark UI and execution plan analysis
PySpark Production Readiness
Topics
- Build reliable PySpark jobs: logging, retries, input checks, output checks
- Package reusable code: modules, configs, parameters, environments
- Monitor production pipelines: failures, duration, volume, alerts
Machine Learning with PySpark MLlib
Topics
- Explore MLlib: ML basics, use cases
- Prepare features: encode, scale, assemble
- Build pipelines: transformers, estimators, stages
- Train models: classification, regression
- Evaluate models: metrics, validation, prediction
Databricks and the Lakehouse Platform
Topics
- Platform basics and workspace areas
- Lake, warehouse and lakehouse concepts
- Databricks use cases
Workspace, Notebooks and Collaboration
Topics
- Navigate the workspace: folders and Repos
- Create, run and debug notebooks
- Python, SQL and Markdown cells
- Share notebooks
Compute Clusters and Runtime
Topics
- Cluster types and node options
- Databricks Runtime basics and libraries
- Attach notebooks and control costs
Data Ingestion and File Handling
Topics
- Access files: upload and mounts
- Read CSV, JSON and Parquet
- Tables, schemas and metadata; ingest to table
Delta Lake Fundamentals
Topics
- Delta basics and ACID transactions
- Create, update and query Delta tables
- Schema enforcement and evolution
- Time travel
Transformation and the Medallion Architecture
Topics
- Filter, select and sort
- Clean nulls, types and duplicates
- Joins and GroupBy aggregations
- Bronze, Silver and Gold layers
Databricks SQL and Dashboards
Topics
- SQL warehouses and queries
- SELECT, WHERE and GROUP BY
- Charts and dashboards
Jobs, Workflows and Governance
Topics
- Notebook jobs and tasks
- Schedules and monitoring
- Unity Catalog: catalogs, schemas and tables
- Manage access