Data Pipelines and Orchestration

Understand the building blocks of a data pipeline and the design choices that decide whether it stays reliable.

• Sources, sinks, and the extract-transform-load lifecycle
• ETL versus ELT and where transformation should run
• Batch versus micro-batch versus streaming at a high level
• Raw, staged, and curated data layers (medallion-style layouts)
• Configuration, secrets, and environment separation
• Failure modes: late data, schema drift, and partial loads

Hands-on lab: Sketch a pipeline for a chosen source system, then implement a minimal extract-and-load script in Python that reads from a REST API and lands raw JSON files.

Write extraction and transformation code that is testable, observable, and resilient to upstream problems.

• Reading from APIs, databases, and files with pagination and rate limits
• Retries with exponential backoff
• Transformations with pandas and typed validation models
• Structured logging and meaningful error messages
• Unit testing transformation functions with pytest
• Packaging pipeline code as an installable module

Hands-on lab: Refactor a single script into small testable functions, add retry logic and pytest cases for malformed records, and run the suite locally.

Load only what changed and make every run safe to repeat.

• Full refresh versus incremental loading
• High-watermark columns and their weaknesses
• Change data capture concepts at a high level
• MERGE and upsert patterns
• Delete-and-insert by partition as an idempotent strategy
• Handling late-arriving and duplicate records

Hands-on lab: Implement a watermark-based incremental load into Postgres, run it three times with overlapping data, and prove with a query that no duplicates appear.

Schedule, order, and recover workflows using Airflow's core concepts.

• DAGs, tasks, operators, and the TaskFlow API
• Schedules, logical dates, and data intervals
• Dependencies, trigger rules, and branching
• Retries, timeouts, and failure callbacks
• Sensors and dataset-driven scheduling
• Variables, connections, and managing secrets

Hands-on lab: Run Airflow locally in Docker, convert your extract-and-load script into a three-task DAG with retries, and trigger it for a past date range.

Store data in open, efficient formats and understand what table formats add on top of files.

• Object storage layout and partitioning conventions on Amazon S3 and similar services
• Row-oriented versus columnar formats: CSV, JSON, Avro, and Parquet
• Parquet row groups, compression, and predicate pushdown
• What Delta Lake and Apache Iceberg add: ACID transactions, schema evolution, and time travel
• Small-file problems and compaction
• Choosing a table format and catalog

Hands-on lab: Write the same dataset as CSV and partitioned Parquet, compare size and query time, then create a Delta or Iceberg table and use time travel to read a previous version.

Pair orchestration with in-warehouse transformation, and prepare the result for deployment.

• Loading raw data first, transforming inside the warehouse
• Triggering dbt runs from Airflow
• Incremental dbt models and full-refresh handling
• Backfills and parameterized runs
• Deploying DAGs through CI and promoting from development to production
• Managed Airflow options such as Amazon MWAA and Cloud Composer

Hands-on lab: Extend your DAG so that after loading raw data it runs a dbt project that builds an incremental model, perform a backfill across a week of dates, and add a CI check that validates the DAG imports cleanly.

Make pipeline health visible and catch data problems before consumers do.

• Data quality checks: row counts, nulls, uniqueness, and freshness
• Great Expectations and dbt tests compared
• Pipeline unit and integration testing strategies
• Lineage concepts and OpenLineage events
• Metrics, logs, and alerting for failed or slow runs
• Service level objectives for data freshness and runbooks for on-call response

Hands-on lab: Add data quality checks that fail a DAG on bad input, wire a failure alert, and inspect the lineage graph of your pipeline.