Capability Development / Data Engineering
Data Engineering on Microsoft Azure (DP-203)
This training program is designed to build practical data engineering capabilities on Microsoft Azure, focusing on data ingestion, transformation, storage, orchestration, security, monitoring and optimization using services such as Azure Data Lake Storage Gen2, Azure Data Factory, Azure Synapse Analytics, Azure Databricks, Azure Stream Analytics and Microsoft Purview. Learners design and implement secure, scalable data storage solutions, build reliable batch and streaming pipelines, and transform data using SQL and Apache Spark. The course closes with a capstone project and assessment-readiness activities that prepare learners for DP-203-aligned skills through practical scenarios, labs and case studies.
Program Overview
This training program is designed to build practical data engineering capabilities on Microsoft Azure, focusing on data ingestion, transformation, storage, orchestration, security, monitoring and optimization using services such as Azure Data Lake Storage Gen2, Azure Data Factory, Azure Synapse Analytics, Azure Databricks, Azure Stream Analytics and Microsoft Purview. Learners design and implement secure, scalable data storage solutions, build reliable batch and streaming pipelines, and transform data using SQL and Apache Spark. The course closes with a capstone project and assessment-readiness activities that prepare learners for DP-203-aligned skills through practical scenarios, labs and case studies.
Audience Profile
Data engineers and database professionals; BI developers and analytics professionals; Cloud engineers working with Azure data platforms; Learners preparing for Azure data engineering roles; and Professionals with basic cloud computing, SQL and Python knowledge.
Course Syllabus
Introduction to Azure Data Engineering
Topics
- Azure Data Engineer Role: responsibilities in modern cloud data projects, core data engineering skills, cloud-based analytics solution ownership
- Modern Data Platforms: analytics architecture, data lake and data warehouse concepts, lakehouse architecture fundamentals
- Processing Patterns: batch processing fundamentals, stream processing concepts, real-time analytics use cases
- Azure Data Services: key Azure data services overview, DP-203-aligned skill areas, service selection for data engineering scenarios
Design and Implement Data Storage
Topics
- Azure Data Lake Storage Gen2: containers and directories, hierarchical namespace, access tiers and storage optimization
- Data Lake Design: raw layer structure, cleansed and curated layers, serving layer organization
- Partitioning Strategy: performance improvement, scalable data organization, cost-optimized partition planning
- File Formats: CSV and JSON, Parquet and Avro, Delta Lake usage scenarios
- Azure Synapse Storage: dedicated SQL pool basics, enterprise data warehousing foundations, storage design considerations
Data Ingestion and Orchestration
Topics
- Pipeline Architecture: Azure Data Factory components, Synapse Pipelines components, common orchestration use cases
- Connectivity Setup: linked services, datasets, integration runtimes, secure connections
- Pipeline Activities: copy activity, mapping data flows, triggers, parameters and variables
- Incremental Loading: metadata-driven pipelines, watermarking techniques, error handling patterns
- Pipeline Operations: monitoring and logging, retry policies, operational best practices
Data Transformation with SQL and Spark
Topics
- Serverless SQL Pools: querying files from Azure Data Lake, using SQL for lake-based exploration, external data access patterns
- T-SQL Transformations: views, external tables, CETAS, reusable transformation logic
- Apache Spark Fundamentals: distributed processing concepts, large-scale workload execution, Spark-based data transformation
- Synapse Spark Notebooks: exploratory analysis, data transformation, collaborative development
- Delta Lake Implementation: transactional data lake patterns, reliability and consistency, Delta table usage
Azure Databricks for Data Engineering
Topics
- Databricks Workspace: clusters, notebooks, jobs, collaborative development environment
- DataFrames and Spark SQL: reading structured data, writing transformed data, querying and transforming data
- Data Transformation Workflows: data cleansing, data enrichment, aggregation, engineering-ready outputs
- Delta Lake Features: Delta tables, schema evolution, time travel, optimization commands
- ADF Integration: connecting Databricks with Azure Data Factory, integrating with data lake storage, automating Databricks jobs
Real-Time and Streaming Data Processing
Topics
- Streaming Architecture: event-driven processing, real-time ingestion, analytics patterns
- Streaming Sources: Azure Event Hubs, IoT Hub, streaming ingestion scenarios
- Azure Stream Analytics: jobs, inputs and outputs, query language fundamentals
- Stream Processing Logic: windowing, aggregations, joins, late-arriving event handling
- Throughput Monitoring: workload monitoring, scaling, performance management
Security, Governance, and Compliance
Topics
- Identity and Access: Microsoft Entra ID, managed identities, authentication options
- Authorization Controls: role-based access control, access control lists, secure data lake access
- Data Protection: encryption, network security, private endpoints, key management
- Microsoft Purview: data discovery, cataloging and lineage, classification and governance
- Compliance Controls: data retention, privacy, auditability, regulatory alignment
Monitoring, Optimization, and Troubleshooting
Topics
- Azure Monitoring: Azure Monitor, Log Analytics, pipeline run history, workload visibility
- Bottleneck Analysis: storage diagnostics, pipeline diagnostics, Spark job diagnostics, SQL pool and streaming job diagnostics
- Performance Optimization: partitioning, file sizing, indexing and caching, query tuning
- Cost Optimization: compute cost control, storage cost control, orchestration cost-control strategies
- Troubleshooting: data quality issues, pipeline failures, authentication issues, performance issue resolution
Capstone Project and Assessment Readiness
Topics
- End-to-End Solution Design: complete Azure data engineering solution, ingestion layer design, serving layer design
- Batch Pipeline Project: transformation workflows, validation steps, loading workflows
- Streaming Pipeline Project: real-time analytics scenarios, streaming service implementation, operational monitoring
- Best-Practice Application: security practices, governance practices, monitoring and optimization practices
- Assessment Readiness: DP-203-style case scenarios, practice questions, readiness checklist