Build a Foundation for Azure Data Engineering Career
What you'll learn
Azure Fundamentals For Data Engineering viz. Azure Active Directory, Key Vault, Storage, Data Lake
Programming with Azure SQLDB Using T-SQL
Azure Data Factory Pipelines, Datasets, Triggers, Global Variables, Linked Services
Azure Active Directory Users, Groups, AD Roles, RBAC Roles
Implement AAD Custom Roles Similar To Real World Projects
Dynamic Pipeline Execution
Configure AAD Users & Groups Access in Azure SQLDB
Build Metadata Driven ETL Orchestration Framework Using Azure SQLDB And Data Factory
Requirements
No Azure background is required. You will learn everything you need to know.
No Programming background is required. T-SQL Programming is covered including Stored Procs, Functions, Cursors, SQL Dynamic Execution, Transactions.
Basics of Azure SQLDB is covered.
Azure Data Factory setup is covered, no prior knowledge is necessary.
Azure Data Factory is covered from scratch.
An end to end ETL Metadata driven Orchestration framework is designed & developed so that learners can get insights in ETL Design & Approach.
Description
We have launched an in-depth technical course for learning Azure Data Factory (ADF) & Azure SQL. There are several other online courses that cover Azure Data Factory. So what is different in this course and why you should consider enrolling in this course?Here are the key differentiators.Unlike other courses, we recognize that ADF cannot be learnt in a silo. One must know Azure Fundamentals, a few key Azure Services and in many instances, knowledge of Azure SQLDB & T-SQL is also necessary. At a minimum, one must know, Azure Active Directory, Azure Storage & Azure Key Vault. Knowledge of these 3 Azure Fundamental Services is required by all Azure Data professionals.The focus of this course is to provide a well rounded & structured step by step training in all the areas that we have mentioned above. The learnings & knowledge gained in this course can be considered as a foundational step in starting your journey towards becoming an Azure Data Engineer. After covering the technical knowledge areas, we go on to develop a metadata driven ETL Orchestration Framework. The aim is to help you build architectural & design skills so that you can grow in your career as an Azure Data Architect or Azure Solution Architect. We welcome you to this course & hope that you gain useful skills to solve complex problems. Our best wishes on your learning journey!
Overview
Section 1: Azure Fundamentals
Lecture 1 Introduction To The Course & Important Note On Volume Control
Lecture 2 Evolution of Data Processing Applications
Lecture 3 Azure Fundamental Concepts
Lecture 4 Sign-up For Azure Free Account
Lecture 5 Upgrade to a Pay-As-You-Go Account
Lecture 6 Azure Portal Overview
Lecture 7 Setting A Budget
Lecture 8 Introduction To Azure Active Directory & Users
Lecture 9 Azure Active Directory Roles (AD Roles, RBAC Roles & Custom Roles)
Lecture 10 Azure Key Vault
Lecture 11 Lab - Azure Active Directory & Key Vault (Part 1)
Lecture 12 Lab - Azure Active Directory & Key Vault (Part 2)
Lecture 13 Azure Storage Accounts
Lecture 14 Lab - Azure Storage Accounts (Part 1)
Lecture 15 Lab - Azure Storage Accounts (Part 2)
Lecture 16 Azure Data Lake
Section 2: Azure SQL Database & T-SQL
Lecture 17 Introduction To Azure SQL Offerings (Part 1)
Lecture 18 Introduction To Azure SQL Offerings (Part 2)
Lecture 19 Lab - Provision Azure SQL Database
Lecture 20 Lab - Install SQL Server Management Studio
Lecture 21 Azure SQL Database Schema/Tables/Logins/Users/Roles/Permissions
Lecture 22 Lab - Azure SQL Database Logins/Users/Permissions
Lecture 23 Lab Script - Azure SQL Database Logins/Users/Permissions
Lecture 24 Azure SQL Database Tables
Lecture 25 Lab - Azure SQL Database Tables
Lecture 26 Lab Script - Azure SQL Database Tables
Lecture 27 Azure SQL Database Indexes With Lab
Lecture 28 Lab Script - Azure SQL Database Indexes
Lecture 29 Azure SQL Database Table Constraints With Lab
Lecture 30 Lab Script - Azure SQL Database Table Constraints
Lecture 31 Azure SQL Database SELECT With Lab
Lecture 32 Lab Script - Azure SQL Database SELECT
Lecture 33 Azure SQL Database Table Joins With Lab
Lecture 34 Lab Script - Azure SQL Database Table Joins
Lecture 35 Azure SQL Database Views With Lab
Lecture 36 Lab Script - Azure SQL Database Views
Lecture 37 Introduction To Azure T-SQL With Lab (Part 1)
Lecture 38 Introduction To Azure T-SQL With Lab (Part 2)
Lecture 39 Lab Script - Introduction To Azure T-SQL
Lecture 40 Azure SQL Cursors With Lab
Lecture 41 Lab Script - Azure SQL Cursors
Lecture 42 Azure SQL Transactions With Lab
Lecture 43 Lab Script - Azure SQL Transactions
Lecture 44 Azure SQL Database Stored Procedures With Lab
Lecture 45 Lab Script - Azure SQL Database Stored Procedures
Lecture 46 Azure SQL Database User Defined Functions With Lab
Lecture 47 Lab Script - Azure SQL Database User Defined Functions
Section 3: Azure Data Factory
Lecture 48 ADF Overview
Lecture 49 ADF Provisioning
Lecture 50 Linked Services
Lecture 51 First Pipeline - Lookup / Set Variable / Datasets
Lecture 52 Foreach Activity - Processing Items In A Loop
Lecture 53 Using Stored Procedures With Lookup Activity
Lecture 54 Read File/Folder Properties Using Get Metadata Activity
Lecture 55 Validation Activity Vs Get Metadata Activity
Lecture 56 Conditional Execution Using IF Activity
Lecture 57 Design & Build First Metadata Driven ETL Framework (Part 1)
Lecture 58 Design & Build First Metadata Driven ETL Framework (Part 2)
Lecture 59 Using Wait Activity As A Timer
Lecture 60 Using Fail Activity To Raise Exceptions
Lecture 61 Using Append Activity With Array Variables
Lecture 62 Using Filter Activity To Selectively Process Files
Lecture 63 Using Delete Activity To Cleanup Files After Processing
Lecture 64 Copy A Single JSON File
Lecture 65 Copy A Single TEXT File
Lecture 66 Copy A Single PARQUET File
Lecture 67 Copy All Files In A Folder
Lecture 68 Binary File Copy & Implementing File Move
Lecture 69 File To Table Copy Using Insert & Upsert Techniques
Lecture 70 File To Table Copy Using Stored Procedure
Lecture 71 File To Table Copy - Large File Issue
Lecture 72 Table To File Copy
Lecture 73 Table To Table Copy
Lecture 74 Master-Child Pattern Using Execute Pipeline Activity
Lecture 75 Implement WHEN-THEN Logic Using Switch Activity
Lecture 76 Iterative Execution Using Until Activity
Lecture 77 Consume REST API Using Web Activity As Well As Copy Activity
Lecture 78 Persist REST API Consumed Data To a File In Azure Storage
Lecture 79 Provision Databricks & Use Databricks Notebook Activity
Lecture 80 Using Self Hosted Integration Runtime To Ingest On-Prem Data Into Azure
Lecture 81 Parameterized Linked Service & ADF Global Parameters
Lecture 82 Automated Execution Of Pipeline Using Scheduled Triggers
Lecture 83 Event-Based Execution Of Pipeline Using Event Triggers
Lecture 84 ADF Limitation - No Iterative Activity Within Foreach Activity
Lecture 85 ADF Limitation - No Iterative Activity Within IF Activity
Lecture 86 ADF Limitation - No Iterative Activity Within SWITCH Activity
Lecture 87 Sequential Vs Parallel Batch In A Foreach Activity
Lecture 88 ADF Limitation - Dynamic Execution Of Pipeline Not Supported In Execute Pipeline
Lecture 89 ADF Limitation - Record Number & Data Size Restrictions In Lookup Activity
Lecture 90 ADF Limitation - Dynamic Variable Name In Set Variable Activity Not Supported
Lecture 91 ADF Overall Limitations
Section 4: Project - Metadata Driven ETL Framework
Lecture 92 Set Up Azure Active Directory Users/Groups & Key Vault
Lecture 93 Set Up Azure Storage
Lecture 94 Set Up Azure SQL Database
Lecture 95 Set Up Additional Groups & Users
Lecture 96 ETL Framework Metadata Tables
Lecture 97 Lab Script - ETL Framework Metadata Tables
Lecture 98 Set Up Azure Data Factory
Lecture 99 Modular & Reusable Design
Lecture 100 Generic Pipeline to Extract From SQL Database (Part 1)
Lecture 101 Generic Pipeline to Extract From SQL Database (Part 2)
Lecture 102 Generic Pipeline to Extract From SQL Database (Part 3)
Lecture 103 Generic Pipeline to Extract From SQL Database (Part 4)
Lecture 104 Lab Script - Metadata Table Configurations
Lecture 105 Lab Script - sp_task_parameters
Lecture 106 Lab Script - sp_job_task_log
Lecture 107 Lab Script - sp_update_job_control
Lecture 108 Lab Script - sp_get_last_run_date
Lecture 109 Lab Script - sp_update_last_run_date
Lecture 110 Generic Pipeline to Load Extracted Data Into Staging Table
Lecture 111 Lab Script - Metadata Table Configurations
Lecture 112 Lab Script - Database Configurations For Bulk Loads
Lecture 113 Lab Script - Customer_Target_Stg Table
Lecture 114 Lab Script - sp_load_stg
Lecture 115 Generic Pipeline to Load Data From Staging Table Into Target Table
Lecture 116 Lab Script - Metadata Table Configurations
Lecture 117 Lab Script - Customer_Target Table
Lecture 118 Lab Script - sp_load_customer
Lecture 119 Job Orchestration Pipeline (Part 1)
Lecture 120 Job Orchestration Pipeline (Part 2)
Lecture 121 Lab Script - Environment Parameters
Lecture 122 Lab Script - sp_get_tasks
Lecture 123 Lab Script - sp_init_job_control
Lecture 124 Lab Script - sp_job_status_update
Lecture 125 Lab Script - sp_task_status
Developers who want to learn Azure Data Factory and Azure SQLDB with no prior knowledge of Azure.,Azure Data Engineers who want to advance their knowledge and learn to design & build professional Azure ETL pipelines.,Azure Data Factory developers who want to learn Azure SQLDB and use it extensively in Data Factory projects
Screenshots