Scalable ML Pipelines with Airflow
DAG-based pipelines, task dependencies, data ingestion → training → evaluation → deploy.
Scalable ML Pipelines with Airflow is a free Learn AI with Python lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Learn AI with Python learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why Orchestration
An ML workflow has many steps: ingest data, build features, train, evaluate, deploy. Running these by hand is fragile. Apache Airflow orchestrates them as code, with scheduling, retries, dependencies, and monitoring built in.
The DAG
Airflow models a pipeline as a DAG (Directed Acyclic Graph) of tasks. Acyclic means no task can depend on itself in a loop, so execution always has a well-defined order from start to finish.
Defining a DAG
You declare a DAG with an id, a schedule, and a start date. The schedule controls how often it runs, for example daily retraining.
from airflow import DAG
import datetime
with DAG(
dag_id="ml_pipeline",
schedule="@daily",
start_date=datetime.datetime(2024, 1, 1),
catchup=False,
) as dag:
...Operators are Tasks
Each node in the DAG is a task created from an operator. Different operators run different kinds of work: Python functions, bash commands, SQL queries, and more.
PythonOperator for Training
The PythonOperator runs a Python callable. It is perfect for a training step that calls your training function.
from airflow.operators.python import PythonOperator
def train_model():
# load features, fit model, save artifact
...
train = PythonOperator(
task_id="train",
python_callable=train_model,
)BashOperator for Evaluation
The BashOperator runs a shell command, handy for invoking an evaluation script or CLI tool.
from airflow.operators.bash import BashOperator
evaluate = BashOperator(
task_id="evaluate",
bash_command="python /opt/ml/evaluate.py --model latest",
)Defining Dependencies
The >> operator sets task order: a >> b means b runs after a succeeds. This wires the DAG so evaluation only starts once training finishes.
train >> evaluate
# train must succeed before evaluate runsLonger Dependency Chains
You can chain many tasks to express the full pipeline order. Airflow runs independent branches in parallel and respects every dependency you declare.
ingest >> features >> train >> evaluate >> deployPassing Data with XCom
Tasks run in isolation, so how does one pass a result to the next? XCom (cross-communication) lets a task push a small value (like a model path or a metric) that a downstream task pulls.
def train_model(ti):
path = "/models/run_42.pt"
ti.xcom_push(key="model_path", value=path)
def deploy(ti):
path = ti.xcom_pull(key="model_path", task_ids="train")
# deploy the artifact at pathXCom is for Small Data
XCom is meant for small metadata (paths, ids, metrics), not large datasets. Big artifacts should live in object storage (S3) with only their location passed through XCom. Overusing XCom for large payloads strains the metadata database.
Scheduling and Retries
Airflow adds production robustness for free: the schedule triggers runs automatically, failed tasks retry with backoff, and the UI shows the status of every task and run, with alerting on failure.
Quick Check
Test your Airflow knowledge.
Recap
You learned scalable ML pipelines with Airflow:
- A DAG with a schedule defines the pipeline
- PythonOperator runs training; BashOperator runs evaluation
- The
>>operator sets task dependencies - XCom passes small artifacts between tasks; big data goes to object storage
Frequently asked questions
Is the “Scalable ML Pipelines with Airflow” lesson free?
Yes — the full text of “Scalable ML Pipelines with Airflow” is free to read here on the web, and the Learn AI with Python course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Learn AI with Python course, upgrade to CoddyKit PRO.
What will I learn in “Scalable ML Pipelines with Airflow”?
DAG-based pipelines, task dependencies, data ingestion → training → evaluation → deploy. You practise Learn AI with Python with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Learn AI with Python?
No prior experience is required. Learn AI with Python on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Scalable ML Pipelines with Airflow” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Learn AI with Python lesson?
Yes. Every Learn AI with Python lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- AI System Architecture Patterns
- Scalable ML Pipelines with Airflow
- Feature Stores: Feast and Tecton
- AI System Observability and Monitoring