The DAG Scheduler Model

Workflows as Graphs

Airflow does not treat a pipeline as a script that runs top to bottom.

It treats it as a Directed Acyclic Graph, or DAG. Tasks are nodes. Dependencies are arrows between them.

Directed means each arrow points one way. Acyclic means no arrow ever loops back.

That second rule matters: a cycle would leave the scheduler waiting forever.

Notice that every arrow in the valid graph moves forward, while the rejected version loops back on itself and could never finish.

Defined in Python

A DAG is not written in a configuration file. It is written in Python code.

This means pipelines can use loops, functions, and conditionals to build tasks dynamically.

Your Python file is a definition, not a run. Airflow reads it to learn the shape of the graph.

The actual execution happens later, and it is the scheduler that decides when.

How the Scheduler Dispatches

The scheduler is the part of Airflow that keeps asking one question: what is ready to run right now?

It parses the DAG files, checks each task against its dependencies, and looks at what has already finished.

If every upstream task has succeeded, the task becomes eligible and is queued for a worker.

If not, it waits and the scheduler checks again on its next pass.

The Metadata Database

Airflow needs somewhere to remember what happened. That place is the metadata database.

It normally runs on Postgres or MySQL.

Every task state lives there: whether a task is queued, running, succeeded, or failed.

The scheduler reads that stored state to decide the next move, so the database is the shared memory of the whole system.

The Life of a Task

A single task moves through a sequence of recorded states.

It starts as scheduled once its dependencies are met, then becomes queued when handed to a worker.

It turns running while the work happens, and ends as succeeded or failed.

Each change is written back to the metadata database, so a restarted scheduler picks up exactly where things stood.

Three Parts, One System

Airflow's model rests on three pieces that depend on each other.

The DAG in Python describes the shape. The scheduler decides timing. The metadata database holds the truth about state.

Remove any one and the model breaks: no graph means no dependencies, no scheduler means nothing dispatches, no database means no memory.

Why the Rules Hold

The acyclic rule is not a stylistic preference. It is what makes scheduling decidable at all.

Because arrows never loop, there is always a task with no unfinished upstream work, so progress is guaranteed.

Sit with that for a moment before you write your next pipeline.

A workflow graph contains no dependency loop that returns to an earlier task. Is this graph acyclic?

• Acyclic
• Cyclic

Answer: Acyclic

Correct. A DAG must be acyclic, so its dependency arrows cannot form a loop. A cycle could leave the scheduler waiting indefinitely.

A task becomes _____ after its dependencies are met, then _____ for a worker, _____ during execution, and finally _____.

Answer: A task becomes scheduled after its dependencies are met, then queued for a worker, running during execution, and finally succeeded or failed.

The task lifecycle begins when dependencies are met and the task is scheduled. It is then queued for a worker, runs while work is performed, and ends in success or failure.

A task has two upstream tasks: one succeeded and one failed. What should the scheduler do with the downstream task?

• The scheduler leaves the task waiting because an upstream task failed
• The scheduler queues the task because only running upstream tasks matter
• The scheduler runs the task immediately because Python defines it
• The scheduler marks the task succeeded because the database stores it

Answer: The scheduler leaves the task waiting because an upstream task failed

The scheduler checks dependencies and stored task states. A task is eligible only when every upstream task has succeeded; it is then queued for a worker. A task with a failed upstream dependency is not eligible under the stated model.