In this video, we will cover how to automate your Python ETL (Extract, and Load) with Apache Airflow. In this session, we will use the TaskFlow API introduced in Airflow 2.0.

In this tutorial, we will see how to design an ETL Pipeline with Python. We will use SQL Server’s AdventureWorks database running on-premise as a source and load data into Google's BigQuery with Python on clouds. We will cover Google Cloud set to establish connection to BigQuery. We will orchestrate our data pipeline via Apache Airflow.

Link to previous video:    • How to build and automate your Python...  
Link to Pandas video: https://www.youtube.com/watch?v=-jerz...

Link to GitHub repo: https://github.com/hnawaz007/pythonda...

Subscribe to our channel:
   / haqnawaz  

---------------------------------------------
Follow me on social media!

GitHub: https://github.com/hnawaz007
Instagram:   / bi_insights_inc  
LinkedIn:   / haq-nawaz  

---------------------------------------------

#ETL #Python #Airflow

Topics covered in this video:
0:00 - Introduction to Airflow
2:49 - The Setup
3:40 - Script ETL pipeline: Extract
5:52 - Transform
7:39 - Load
8:00 - Define Direct Acyclic Graph (DAG)
9:36 - Airflow UI: Dag enable & run
10:09 - DAG Overview
10:29 - Test ETL Pipeline