Workflows are an essential part of many businesses, whether it’s processing data, automating reports, or orchestrating machine learning pipelines. Managing these workflows manually can be time-consuming and error-prone. This is where Apache Airflow comes in.

Airflow is a powerful, open-source tool that allows you to programmatically create, schedule, and monitor workflows. With Airflow, workflows are defined as code, which means they are easy to manage, version, and share. In this guide, we’ll introduce you to Airflow and walk you through creating your first DAG, the core component of Airflow.

Understanding DAGs in Airflow

A Directed Acyclic Graph (DAG) is the backbone of Airflow. Think of it as a blueprint for a workflow. It defines a series of tasks and the order in which they should be executed.

Each DAG consists of nodes (tasks) and edges (dependencies between tasks). The “acyclic” part means that the workflow cannot have circular dependencies. Tasks must always move forward in a logical sequence.

Example: Data Pipeline

Imagine a simple DAG for processing data:

1. Extract: Fetch raw data from an API.
2. Transform: Clean and format the data.
3. Load: Store the processed data in a database.

This structure ensures that data is extracted before it is transformed and loaded, following a logical order.

Key Concepts in Apache Airflow

1. Tasks

A task represents a unit of work, such as running a Python function, executing a SQL query, or transferring data. Tasks are the building blocks of a DAG.

2. Operators

Operators define what kind of work a task will perform. Some common operators include:

• PythonOperator: Executes a Python function.
• BashOperator: Runs a Bash command.
• EmailOperator: Sends an email notification.
• DummyOperator: Serves as a placeholder (useful for controlling flow).

3. Task Dependencies

Tasks in a DAG must be arranged in a sequence. Airflow allows you to set dependencies using simple syntax:

extract_task >> transform_task >> load_task

This ensures that extract_task runs before transform_task, and transform_task runs before load_task.

4. Task Flow Based on Results

Airflow also allows tasks to branch based on conditions. This is done using the BranchPythonOperator. For example:

from airflow.operators.python_operator import PythonOperator, BranchPythonOperator

def check_data():
    if data_is_valid():
        return 'process_data'
    else:
        return 'send_alert'

check_task = BranchPythonOperator(
    task_id='check_task',
    python_callable=check_data,
    dag=dag,
)

process_data = PythonOperator(
    task_id='process_data',
    python_callable=process_function,
    dag=dag,
)

send_alert = PythonOperator(
    task_id='send_alert',
    python_callable=alert_function,
    dag=dag,
)

check_task >> [process_data, send_alert]

In this example, the workflow follows different paths depending on the result of check_task.

Setting Up Apache Airflow

Before creating a DAG, let’s set up Apache Airflow on your system:

1. Install Apache Airflow

pip install apache-airflow

2. Initialize the Database

Airflow needs a database to store metadata. Run:

airflow db init

3. Start the Web Server

The Airflow web server provides a UI to monitor your DAGs. Start it with:

airflow webserver --port 8080

4. Start the Scheduler

The scheduler is responsible for running tasks at the correct time.

airflow scheduler

Creating Your First DAG

Once Airflow is running, let’s create a simple DAG that prints “Hello, World!”.

1. Create a DAG File

Create a new Python file in the Airflow DAGs directory:

touch ~/airflow/dags/sample_dag.py

2. Define the DAG

Open the file and add the following code:

from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from datetime import datetime, timedelta

def print_hello():
    print("Hello, World!")

# Default arguments
default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'email_on_failure': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}

dag = DAG(
    'sample_dag',
    default_args=default_args,
    description='A simple Hello World DAG',
    schedule_interval=timedelta(days=1),
    start_date=datetime(2023, 1, 1),
    catchup=False,
)

hello_task = PythonOperator(
    task_id='hello_task',
    python_callable=print_hello,
    dag=dag,
)

3. Save and Deploy the DAG

Ensure this file is saved in the ~/airflow/dags/ directory. The scheduler will detect it automatically.

4. Access the Airflow UI

Visit http://localhost:8080 in your browser. You should see sample_dag listed.

5. Trigger the DAG

Click on sample_dag, then click “Trigger DAG” to execute it.

Conclusion

Airflow is a fantastic tool for automating workflows, making them scalable, maintainable, and easy to monitor. By defining workflows as code, you can version-control your processes and integrate them with other systems.

In this guide, we introduced key Airflow concepts, walked through setting up Airflow, and created a simple DAG. As you become more comfortable with Airflow, you’ll be able to design complex, dynamic workflows that adapt based on real-time data.