Scheduled and Periodic Jobs with Celery Beat

Your FastAPI app fires off one-off background tasks with Celery just fine, but some work must run on a schedule: send a digest email every morning, expire abandoned carts every 10 minutes, recompute analytics nightly.

Celery Beat is a scheduler process. It does not execute tasks itself — it wakes up on a tick, decides which tasks are due, and pushes them onto the broker (Redis/RabbitMQ). Your normal celery worker processes pick them up and run them.

• Beat = the clock that publishes due tasks.
• Worker = the muscle that runs them.

This separation is the whole reason periodic jobs scale: one Beat, many workers.

Schedules live on the Celery app under conf.beat_schedule. Each entry maps a name to a dict with the task path, a schedule (seconds, timedelta, or a crontab), and optional args/kwargs.

The simplest schedule is a fixed interval. Below, the cleanup task runs every 30 seconds. The task path string must match the worker's registered name exactly.

Fixed intervals are coarse. For real calendar logic — "every weekday at 07:30" — use celery.schedules.crontab. It mirrors Unix cron fields: minute, hour, day_of_week, day_of_month, month_of_year.

• crontab(minute=0, hour=0) — midnight every day.
• crontab(minute="*/15") — every 15 minutes.
• crontab(hour=7, minute=30, day_of_week="1-5") — 07:30 Mon–Fri.

Unset fields default to * (every value), exactly like a crontab line.

Each schedule entry can carry arguments and per-call options. Use args (positional) or kwargs (keyword) to parametrize the same task differently across entries. The options dict lets you route a periodic task to a specific queue, set expires, or override priority.

expires is important for periodic jobs: if Beat enqueues a task but workers are backed up, an expired message is discarded instead of running late and stacking up.

Instead of one giant beat_schedule dict, you can register entries near the tasks using the on_after_configure signal and app.add_periodic_task. This keeps the schedule co-located with the code it triggers and avoids stale string paths.

add_periodic_task(schedule, signature, name=...) takes the interval or crontab first, then the task signature. You can pass task.s(arg) to bake in arguments.

By default Celery interprets crontab times in UTC. If you write crontab(hour=7) expecting local 07:00, your job will fire at the wrong wall-clock time. Always set the timezone explicitly and decide deliberately.

• timezone — the zone crontab times are evaluated in.
• enable_utc=True — keep internal timestamps in UTC (recommended) while still evaluating crontabs in your chosen timezone.

Pin the timezone in config so every developer and every server agrees, regardless of the host's local TZ.

You start the scheduler as its own process. The default scheduler stores its "last run" state in a small file so it does not re-fire everything after a restart.

• celery -A app beat -l info — run Beat.
• --schedule /var/run/celerybeat-schedule — where the shelve state file lives.
• You can run worker + beat together in dev with celery -A app worker -B, but never use -B in production — it ties the clock to one worker's lifecycle.

In production, run exactly one Beat process. Two Beats = every periodic task fires twice.

You can scale workers horizontally to dozens of pods and Celery handles it: the broker distributes each queued message to exactly one worker. But Beat is the clock, and you must run only one clock.

If two Beat processes run, each independently decides a task is due and publishes it, so subscribers see duplicate executions. Common ways this happens by accident:

• Two replicas of a pod that both launch beat.
• Using worker -B and then scaling that worker deployment to 2+.

Fix: a dedicated Beat Deployment with replicas: 1, separate from the worker Deployment you scale freely.

Even with one Beat, duplicates happen — a Beat restart at the wrong second, an at-least-once broker redelivery, or an operator mistake. The robust defense is making periodic tasks idempotent: running twice has the same effect as running once.

A simple pattern is a short-lived distributed lock in Redis using SET NX with an expiry. Whoever acquires the lock does the work; concurrent or duplicate runs skip safely.

The default file scheduler means changing a schedule requires editing code and restarting Beat. For dynamic schedules you want a backend store you can edit at runtime.

• RedBeat — stores schedules in Redis; great for a Redis-only FastAPI stack. Set beat_scheduler = "redbeat.RedBeatScheduler".
• django-celery-beat — stores entries in SQL tables, editable via admin UI.

RedBeat also provides a Redis-based lock so that if you accidentally start two Beats, only one is the active scheduler — a safety net for the single-clock rule.

To build intuition for what Beat does on each tick, here is a tiny standalone simulation: given a list of cron-like jobs (interval in seconds and a last-run timestamp), decide which are due now. This is the core loop Beat performs — compare "now" against "next due" — without any broker.

You deploy a FastAPI + Celery stack to Kubernetes. To handle load you scale your worker Deployment to 4 replicas, and each worker is started with celery -A app worker -B. Periodic tasks start running 4 times each. What is the correct fix?

You can now schedule recurring work with Celery Beat:

• Beat schedules, workers execute — Beat publishes due tasks to the broker; one Beat, many workers.
• Schedules live in beat_schedule or via add_periodic_task; use timedelta for intervals and crontab() for calendar logic.
• Per-entry args/options parametrize tasks and set queue/expires to avoid stale pile-ups.
• Timezones are UTC by default — pin timezone + enable_utc explicitly.
• One clock only — never worker -B in production; run a single dedicated Beat replica.
• Idempotency (Redis SET NX locks) protects against duplicate or redelivered runs.
• RedBeat / django-celery-beat give editable, persistent, lock-protected schedules at runtime.