Azure cloud introduction
* Azure data engineer products.
* introduction to data warehouse
* what is etl and elt
* azure data engineer products
* introduction to azure sql
* introduction to azure synapse analytics
* introduction to azure data lake
* different varieties of data
* different formats of data
* introduction to Azure data factory.
* Adf as etl or elt ? when to use what?
* Adf version 1 and 2. what are additional features into adf
* introduction to azure data bricks
* difference between databricks and azure databricks
* what is spark
* introduction to spark components:
spark core,
spark sql,
spark mllib,
spark graphx,
spark streaming
* programming languages used for spark coding in Azure data bricks:
Python,
R,
SQL,
Scala
* introduction to Azure streaming
* what is micro batching
Introduction to Nosql databases
* Nosql database types:
Key value databases,
Columner databases,
Document databases,
Graph databases
* limitations of rdbms databases,
and nosql databases will solve problems.
* When to use what type of nosql database
* introduction to Azure cosmos db
* introduction to Azure Key vault.
* Data engineer roles , resposibilities and challenges.
* Deeper look into Azure data factory.
Building Blocks of ADF
* what are pipelines
* what are activities in pipelines
* Integration Runtime and its importance
* types of Integration Runtimes:
Auto Integration Runtime,
Self-hosted Integration Runtime,
SSIS Integration Runtime
* When to use what type of Integration runtime.
* Linked services introduction
* how many linked services are required? how to decide?
* Datasets introduction
Introduction to dataflows.
* Azure regions and importance
* Lab: Azure UI
* Lab: Azure resources and resource groups
* Lab: create resource group.
* Lab: create data lake gen2 storage account
* Lab: create containers in datalake
* Lab: upload local files into datalake containers
* Lab: azure sql database and server instance
* Lab: connect Azure sql with Query editor
* Lab: connection error: firewall issue
* Lab: Adding sql client ip to firewall rules.
* Lab: creating adf instance
* Lab: launching adf instance
* Lab: understanding adf User Interface
* Lab: creating Linked service for data lake gen2 storage account.
* Lab: Linked service connection error for data lake.
* Lab: How to solve Linked service connection error for data lake.
* what are soft delete options for Blobs and containers.
* permission errors while creating linked service for datalake
* Lab: how to assign “Storage Blob data Contributor” role to users/groups
* Lab: creating datasets for Azure Data lake.
* copy activity
* Lab: creating adf pipeline to copy data from one container to other container of same storage account of data lake with copy activity

* Lab: creating pipeline to copy data from one storage account to other storage account with copy
activity
* Lab: creating linked service for Azure Sql database
* Lab: connection errors while creating linked service
* Lab: solving connection error for azure sql linked service
* Lab: creating dataset for azure sql table
* Lab: creating adf pipeline to load data from Azure datalake to Azure Sql table
* Lab: loading errors
* Lab: solving loading errors.
* Lab: get meta data activity
* how to interpret output of get meta data activity
* Lab: filter activity
* Lab: foreach activity
* Lab: Wait activity
* Lab: if else activity
* Lab: lookup activity
* Lab: Append Activity
* what are dataflows and deeper look into dataflows.
* understanding dataflow transformations
* Lab: Dataflow Source tranformation
* Lab: Dataflow sink transformation
* Lab: loading data from datalake to datalake using Dataflows.
* Lab: loading data from datalake to Azure sql table using Dataflows.
* Lab: loading data from datalake to Azure Synapse using Dataflows
* Lab: loading data from Azure sql to Azure synapse using dataflows
* Lab: conditional split using Dataflows
* Lab: Derived Column transformation in Dataflows.
* Lab: Lookup transformation in dataflows.
* Lab: Select transformation in dataflows.
* Lab: Filter transformation in dataflows.
* Lab: Join tranformation in dataflows.
* types of joins
* Lab: “Exists” tranformation in dataflows.
* types of “Exists” operations.
* Lab: “Union” tranformation in dataflows.
* Lab: “Aggregate” transformation in dataflows.
* Lab: “Surrogate Key” transformation in dataflows.
* Lab: “Pivot” transformation in dataflows.
* Lab: “Unpivot” transformation in dataflows.
* Lab: “Window” transformation in dataflows.
* Lab: “Rank” tranformation in dataflows.
* Lab: “Flatten” transformation in dataflows.
* Lab: “Parse” transformation in dataflows.
* Lab: “stringify” transformation in dataflows.
* Lab: “Sort” transformation in dataflows.
* Lab: “Alter Row” transformation in dataflows.
* introduction to parameterization .
* Lab: parameters for Linked Services
* Lab: parameters for datasets
* Lab: parameters for Pipelines
* Lab: parameters for Dataflows
* Lab: creating Integration runtime : self-hosted
* Lab: creating Linked service for Onpremise database(oracle/mysql)
* Lab: creating pipeline for migrating data from onpremise database to datalake
using copy activity
* Lab: creating pipeline for migrating data from onpremise database to Azure Synapse using copy
activity.
* Lab: creating pipeline for migrating data from onpremise database to datalake
using dataflows.
* Lab: creating pipeline for migrating data from onpremise database to Azure Synapse using dataflows.
* Lab: monitoring pipelines
Introduction to Incremental data load.
* Lab: implementation of Incremental data load with ADF.
* Introduction to SCD( slowly changing dimensions ).
* Lab: SCD implementation with ADF
* Other types of SCD and explanation.
* Lab: SCD type 2 implementation with ADF.
Azure Sql:
* introduction to Azure sql database
* create azure sql database
* create azure sql database server
* types of azure sql instances
* Azure sql virtual machine.
* connect and execute queries on Azure sql
* configure sql client ip address on firewall settings
* Elastic pools
* Active geo replications
* Fail over groups
* dynamic data masking
Azure data lake storage
Azure Synapse Analytics