ETL best practices with Airflow documentation site

Important

Disclaimer: This is not the official documentation site for Apache airflow. This site is not affiliated, monitored or controlled by the official Apache Airflow development effort. If you are looking for the official documentation site, please follow this link:

Official Airflow documentation

What you will find here are interesting examples, usage patterns and ETL principles that I thought are going to help people use airflow to much better effect.

Content

• ETL principles
• Gotcha’s
• What makes Airflow great?
• ETL example
  • Install airflow on host system
  • Run airflow from docker
  • Run it
  • How it works
  • Proof of principles compliance
  • Issues
• Hive example
  • How to start
  • How to run
  • How it works
  • Strategy
• Data vault
  • Methodology
  • How to start the example
  • Limitations
• Monitoring
  • Data quality monitoring
  • Taking it a step further
  • Beyond detection
• Building your own ETL platform
  • Understanding the airflow platform design
  • Understanding hooks and operators
  • Publish documentation
  • Strategies for testing the platform
• Ingesting files
  • Extending from the example
• Tips
  • Long running processes
  • Quick deployment
  • Connection administration
  • Implement a “Functional ETL” approach
  • Datavault
• Deployments
  • Template and deploy managed services
  • Continuous integration
  • Log files
• Data Vault 2
  • About Datavault
  • Overall flow
  • Staging flow
  • Data vault loading flow
  • End dating
  • Star Schema
  • Future considerations
  • Data issues
• Data Vault with Big Data processes
  • 1. Extract data
  • 2. Process full dumps
  • 3. Process incremental data (PSA)
  • 4. Loading into Hive
  • Principles
  • Entities, not records
  • About the implementation
• Data vault with DBT