Course Outline
Introduction to Apache Airflow
- The concept of workflow orchestration
- Principal features and advantages of Apache Airflow
- Enhancements in Airflow 2.x and an overview of the ecosystem
Architecture and Fundamental Concepts
- Scheduler, web server, and worker components
- DAGs, tasks, and operators
- Executors and backends (Local, Celery, Kubernetes)
Installation and Configuration
- Setting up Airflow in local and cloud contexts
- Configuring Airflow with various executors
- Establishing metadata databases and connections
Exploring the Airflow UI and CLI
- Navigating the Airflow web interface
- Tracking DAG runs, tasks, and logs
- Utilizing the Airflow CLI for administrative tasks
Creating and Managing DAGs
- Building DAGs with the TaskFlow API
- Employing operators, sensors, and hooks
- Handling dependencies and scheduling intervals
Integrating Airflow with Data and Cloud Services
- Linking to databases, APIs, and message queues
- Executing ETL pipelines via Airflow
- Cloud integrations: AWS, GCP, and Azure operators
Monitoring and Observability
- Task logs and real-time oversight
- Metrics collection using Prometheus and Grafana
- Setting up alerts and notifications via email or Slack
Enhancing Apache Airflow Security
- Role-based access control (RBAC)
- Authentication through LDAP, OAuth, and SSO
- Secrets management using Vault and cloud secret stores
Scaling Apache Airflow
- Parallelism, concurrency, and task queuing
- Utilizing CeleryExecutor and KubernetesExecutor
- Deploying Airflow on Kubernetes using Helm
Best Practices for Production Environments
- Version control and CI/CD workflows for DAGs
- Testing and debugging DAGs
- Ensuring reliability and performance at scale
Troubleshooting and Optimization
- Resolving issues with failed DAGs and tasks
- Improving DAG performance
- Identifying and avoiding common pitfalls
Recap and Future Steps
Requirements
- Proficiency in Python programming
- Knowledge of data engineering or DevOps principles
- Familiarity with ETL processes or workflow orchestration
Target Audience
- Data scientists
- Data engineers
- DevOps and infrastructure specialists
- Software developers
Testimonials (7)
The instructor adapted the training to the participants’ level and responded to all questions. He was very communicative, and it was easy to interact with him. I really appreciated the format of the training, which included many practical exercises. Overall, it was a very engaging and well-organized session.
Jacek Chlopik - ZAKLAD UBEZPIECZEN SPOLECZNYCH
Course - Apache Airflow: Building and Managing Data Pipelines
The training was spot on. Very useful theory and exercices.
Vladimir - PUBLIC COURSE
Course - Apache Airflow
The training was spot on in all aspects. Usefull theoretical aspects and exercises.
Vladimir - PUBLIC COURSE
Course - Apache Airflow
The training was spot on in all aspects. Usefull theoretical aspects and exercises.
Vladimir - PUBLIC COURSE
Course - Apache Airflow
The training was spot on in all aspects. Usefull theoretical aspects and exercises.
Vladimir - PUBLIC COURSE
Course - Apache Airflow
The training was spot on in all aspects. Usefull theoretical aspects and exercises.
Vladimir - PUBLIC COURSE
Course - Apache Airflow
The training was spot on in all aspects. Usefull theoretical aspects and exercises.
