Machine Learning Academy · Lesson

Automated Retraining Pipelines with GitHub Actions

Learners will write a GitHub Actions workflow that triggers on a data schedule, runs training, evaluates the new model, and promotes it only when it beats the production baseline.

Automated Retraining Pipelines with GitHub Actions is a free Machine Learning Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Automate Model Retraining?

Models trained on historical data degrade over time as real-world patterns shift. Manually retraining requires a data scientist to remember to do it, run scripts, evaluate results, and update deployment — a process prone to delays and human error. Automated retraining pipelines run on a schedule or when triggered by data arrival, automatically train a new model, evaluate it against the current champion, and promote it only if performance improves — with zero manual intervention.

GitHub Actions: Workflows and Triggers

GitHub Actions is a CI/CD platform built into GitHub. Workflows are defined in YAML files under .github/workflows/ and run in response to triggers: code pushes, pull requests, manual dispatches, or cron schedules. Each workflow consists of jobs (groups of steps) that run on hosted or self-hosted runners. For ML pipelines, a scheduled cron trigger on a cloud GPU runner is the most common pattern.

# .github/workflows/retrain.yml (YAML structure shown)
# name: Model Retraining Pipeline
#
# on:
#   schedule:
#     - cron: '0 2 * * 1'  # every Monday at 2 AM UTC
#   workflow_dispatch:       # also allow manual trigger
#
# jobs:
#   retrain:
#     runs-on: ubuntu-latest
#     steps:
#       - uses: actions/checkout@v4
#       - name: Set up Python
#         uses: actions/setup-python@v5
#         with:
#           python-version: '3.11'
print('Workflow YAML structure shown above.')

Cron Syntax for Scheduling

GitHub Actions cron expressions follow standard Unix cron format: minute hour day month weekday. Common ML retraining schedules include weekly retraining (when new weekly data batches are available) and nightly retraining for fast-moving domains. Use workflow_dispatch alongside cron so data scientists can also trigger retraining manually when they push a new training script or detect drift.

# Common cron schedule examples for ML pipelines
# '0 2 * * 1'      -- Every Monday at 02:00 UTC (weekly)
# '0 3 * * *'      -- Every day at 03:00 UTC (nightly)
# '0 */6 * * *'    -- Every 6 hours
# '0 2 1 * *'      -- First day of every month at 02:00

# All times are UTC in GitHub Actions
# Use https://crontab.guru to verify cron expressions

# Example in YAML:
# on:
#   schedule:
#     - cron: '0 3 * * *'   # nightly at 3 AM UTC
#   workflow_dispatch:

print('Cron schedule examples shown above.')

Checkout, Setup Python, Install Deps

The first steps in every ML workflow checkout the repository code, set up the required Python version, and install dependencies from requirements.txt. Use dependency caching with actions/cache to avoid reinstalling packages on every run — this can reduce pipeline duration from 10 minutes to under 2 minutes by reusing the cached pip packages when requirements.txt has not changed.

# .github/workflows/retrain.yml -- steps section
# steps:
#   - uses: actions/checkout@v4
#
#   - name: Set up Python 3.11
#     uses: actions/setup-python@v5
#     with:
#       python-version: '3.11'
#
#   - name: Cache pip packages
#     uses: actions/cache@v4
#     with:
#       path: ~/.cache/pip
#       key: pip-${{ hashFiles('requirements.txt') }}
#
#   - name: Install dependencies
#     run: pip install -r requirements.txt

print('Workflow steps shown above.')

Secrets Management in GitHub Actions

API keys, database passwords, and cloud credentials must never be hard-coded in workflow files. Store them as GitHub Secrets (Settings → Secrets → Actions) and reference them as ${{ secrets.SECRET_NAME }}. They are masked in logs and unavailable to forked pull requests. For ML pipelines, store MLflow credentials, cloud storage keys, and database connection strings as secrets.

# .github/workflows/retrain.yml -- env section with secrets
# jobs:
#   retrain:
#     runs-on: ubuntu-latest
#     env:
#       MLFLOW_TRACKING_URI: ${{ secrets.MLFLOW_TRACKING_URI }}
#       AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
#       AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
#       DATABASE_URL: ${{ secrets.PROD_DATABASE_URL }}
#
# In train.py, read normally:
import os
DB_URL = os.getenv('DATABASE_URL')
MLFLOW_URI = os.getenv('MLFLOW_TRACKING_URI')
print('Config loaded from environment.')

The Training Step

The core step runs your Python training script. The script should accept configuration via environment variables (set in the workflow), log all parameters and metrics to MLflow, and write the new model to the registry as a new version in None stage. Keep the script idempotent: running it twice with the same data should produce a new registry version each time without corrupting existing versions.

# train.py -- called by GitHub Actions
import os
import mlflow
import mlflow.sklearn
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import f1_score

mlflow.set_tracking_uri(os.getenv('MLFLOW_TRACKING_URI', 'http://localhost:5000'))
mlflow.set_experiment('automated_retraining')

with mlflow.start_run(run_name='scheduled_retrain'):
    n_est = int(os.getenv('N_ESTIMATORS', '200'))
    lr = float(os.getenv('LEARNING_RATE', '0.05'))
    mlflow.log_param('n_estimators', n_est)
    mlflow.log_param('learning_rate', lr)

    clf = GradientBoostingClassifier(n_estimators=n_est, learning_rate=lr, random_state=42)
    # clf.fit(X_train, y_train)
    # f1 = f1_score(y_test, clf.predict(X_test))
    # mlflow.log_metric('test_f1', f1)
    # mlflow.sklearn.log_model(clf, 'model', registered_model_name='SentimentClassifier')
    print('Training step complete.')

The Evaluation Step: Beat the Champion

After training, an evaluation step compares the new model against the current Production champion. If the new model wins, it proceeds to promotion; otherwise the pipeline marks the new version as rejected (tag it with status=rejected) and keeps the champion in Production. Use a GitHub Actions output variable to pass the promotion decision between steps.

# evaluate.py -- compare new version against production
from mlflow.tracking import MlflowClient
import sys

client = MlflowClient()
MODEL_NAME = 'SentimentClassifier'

def get_f1(run_id):
    return client.get_run(run_id).data.metrics.get('test_f1', 0)

# Get latest registered version (just added by train.py)
new_versions = client.get_latest_versions(MODEL_NAME, stages=['None'])
new_v = sorted(new_versions, key=lambda v: int(v.version))[-1]
new_f1 = get_f1(new_v.run_id)

prod_versions = client.get_latest_versions(MODEL_NAME, stages=['Production'])
if prod_versions:
    prod_f1 = get_f1(prod_versions[0].run_id)
    print(f'Champion F1: {prod_f1:.4f}  Challenger F1: {new_f1:.4f}')
    should_promote = new_f1 > prod_f1
else:
    print('No champion found.')
    should_promote = True

print('should_promote=' + str(should_promote).lower())
sys.exit(0 if should_promote else 1)  # exit code drives next step

The Promotion Step: Conditional Execution

Use GitHub Actions' if: success() and if: failure() conditions on steps or jobs to implement conditional promotion. After the evaluation step, a promotion step runs only if evaluation succeeded (the challenger won). Use a separate notification step that always runs to send a Slack or email alert with the pipeline outcome, regardless of whether promotion happened.

# promote.py -- transition new version to Production
from mlflow.tracking import MlflowClient

client = MlflowClient()
MODEL_NAME = 'SentimentClassifier'

new_versions = client.get_latest_versions(MODEL_NAME, stages=['None'])
new_v = sorted(new_versions, key=lambda v: int(v.version))[-1]

client.transition_model_version_stage(
    name=MODEL_NAME,
    version=new_v.version,
    stage='Production',
    archive_existing_versions=True
)
print(f'Version {new_v.version} promoted to Production.')

# In the workflow YAML:
# - name: Promote model
#   if: success()  # only runs if evaluate.py exits with 0
#   run: python promote.py

Data Download in the Pipeline

Retraining pipelines need fresh data. Add a data download step before training that fetches the latest data from S3, a database, or a data lake. Use versioned data snapshots (tagged with a date) rather than always pulling the latest to maintain reproducibility: if a model performs poorly, you need to reproduce the exact training data it saw. Log the data version as an MLflow parameter alongside model hyperparameters.

# download_data.py -- fetch latest training data from S3
import boto3
import os
from datetime import date

DATA_DATE = os.getenv('DATA_DATE', str(date.today()))  # e.g., 2024-06-15
BUCKET = os.getenv('S3_BUCKET', 'my-ml-data')
KEY = f'datasets/sentiment/{DATA_DATE}/train.parquet'

s3 = boto3.client('s3')
os.makedirs('/tmp/data', exist_ok=True)
s3.download_file(BUCKET, KEY, '/tmp/data/train.parquet')
print(f'Downloaded data for {DATA_DATE}')

# In train.py, log data_date as a parameter:
# mlflow.log_param('data_date', DATA_DATE)

Notifications with Slack Webhooks

An automated pipeline without notifications is a black box. Add a final notification step that always runs (using if: always()) and posts a summary to Slack or sends an email. Include the run outcome (promoted or rejected), the new version number, the F1 scores of both champion and challenger, and a direct link to the MLflow run for drill-down. Good notifications turn silent automation into a collaborative data science workflow.

import os
import requests
import json

SLACK_WEBHOOK = os.getenv('SLACK_WEBHOOK_URL')
PROMOTED = os.getenv('PROMOTED', 'false') == 'true'
NEW_VERSION = os.getenv('NEW_VERSION', 'N/A')
NEW_F1 = os.getenv('NEW_F1', 'N/A')

status_icon = ':white_check_mark:' if PROMOTED else ':x:'
status_text = 'Promoted to Production' if PROMOTED else 'Champion retained'

message = {
    'text': (f'{status_icon} *ML Retraining Pipeline*\n'
             f'Model: SentimentClassifier v{NEW_VERSION}\n'
             f'Status: {status_text}\n'
             f'New F1: {NEW_F1}')
}
if SLACK_WEBHOOK:
    requests.post(SLACK_WEBHOOK, data=json.dumps(message))
    print('Slack notification sent.')

Complete Workflow YAML Overview

A complete automated retraining workflow has five sequential jobs: download_data (fetch latest dataset), train (run training script, log to MLflow), evaluate (compare vs production champion), promote (transition stage if challenger wins), and notify (always-run Slack alert). Each job passes outputs to the next using GitHub Actions job outputs or shared artifact files. The entire pipeline runs unattended on a weekly schedule.

# Full workflow summary (pseudo-YAML)
# name: Weekly Model Retraining
# on:
#   schedule: [{cron: '0 3 * * 1'}]
#   workflow_dispatch: {}
# jobs:
#   download_data:
#     runs-on: ubuntu-latest
#     steps: [checkout, setup-python, install, run download_data.py]
#   train:
#     needs: download_data
#     steps: [checkout, setup-python, install, run train.py]
#   evaluate:
#     needs: train
#     steps: [checkout, setup-python, install, run evaluate.py]
#   promote:
#     needs: evaluate
#     if: success()
#     steps: [checkout, setup-python, install, run promote.py]
#   notify:
#     needs: [evaluate, promote]
#     if: always()
#     steps: [run notify.py]
print('Full workflow YAML structure shown above.')

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: GitHub Actions cron triggers run retraining on a schedule without manual intervention, evaluation scripts implement the champion/challenger pattern so only better models are promoted, and secrets management keeps credentials out of workflow YAML files using encrypted GitHub Secrets. Next up we explore data drift detection — identifying when the input distribution shifts away from training conditions, which is the most common cause of production model degradation.

Frequently asked questions

Is the “Automated Retraining Pipelines with GitHub Actions” lesson free?

Yes — the full text of “Automated Retraining Pipelines with GitHub Actions” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.

What will I learn in “Automated Retraining Pipelines with GitHub Actions”?

Learners will write a GitHub Actions workflow that triggers on a data schedule, runs training, evaluates the new model, and promotes it only when it beats the production baseline. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Machine Learning Academy?

No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Automated Retraining Pipelines with GitHub Actions” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Machine Learning Academy lesson?

Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

Experiment Tracking with MLflow: Log Params, Metrics, and Artifacts
Reproducible Environments with Docker for ML
Model Registry: Staging, Production, and Archiving
Automated Retraining Pipelines with GitHub Actions

← Back to Machine Learning Academy