DevOps Bootcamp · Lesson

Idempotent Scripts and Retry-with-Backoff Logic

Design operations that are safe to re-run and add exponential backoff for flaky external calls.

Idempotent Scripts and Retry-with-Backoff Logic is a free DevOps Bootcamp lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the DevOps Bootcamp learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

What Is Idempotency and Why It Matters

Idempotency means running the same operation multiple times produces the same result as running it once. In Bash scripting, this is critical because scripts crash, networks drop, and humans re-run things by accident.

A non-idempotent script that creates a user twice may fail or duplicate data.
An idempotent script checks first: does this already exist?
Idempotent scripts are safe to use in cron jobs, CI pipelines, and retry loops.

The golden rule: check before you act. Every destructive or creative operation should be guarded by a precondition test.

Guarding File and Directory Creation

The most common idempotency pattern is checking whether a resource already exists before creating it. Bash provides concise one-liners for this.

[ -d dir ] — true if directory exists
[ -f file ] — true if regular file exists
mkdir -p — creates directory only if absent (built-in idempotency)

Prefer built-in flags like -p and --no-clobber over manual checks when available — they are atomic and race-condition-safe.

#!/usr/bin/env bash
set -euo pipefail

CONFIG_DIR="$HOME/.myapp"
CONFIG_FILE="$CONFIG_DIR/config.ini"

# Idempotent: mkdir -p never fails if dir already exists
mkdir -p "$CONFIG_DIR"

# Idempotent: only write config if it doesn't exist yet
if [ ! -f "$CONFIG_FILE" ]; then
  echo '[defaults]' > "$CONFIG_FILE"
  echo 'timeout=30' >> "$CONFIG_FILE"
  echo "Created $CONFIG_FILE"
else
  echo "Config already exists, skipping."
fi

Idempotent User and Group Management

System administration tasks like adding users or groups must be idempotent — re-running the script on the same machine should not error out or create duplicates.

id username returns 0 if the user exists
getent group groupname checks for a group
Wrap each operation in a guard so the script is safe to replay

This pattern is the foundation of configuration management tools like Ansible — every task is a guarded, idempotent operation.

#!/usr/bin/env bash
set -euo pipefail

APP_USER="apprunner"
APP_GROUP="appgroup"

# Idempotent group creation
if ! getent group "$APP_GROUP" &>/dev/null; then
  groupadd "$APP_GROUP"
  echo "Group '$APP_GROUP' created."
else
  echo "Group '$APP_GROUP' already exists."
fi

# Idempotent user creation
if ! id "$APP_USER" &>/dev/null; then
  useradd -m -g "$APP_GROUP" -s /bin/bash "$APP_USER"
  echo "User '$APP_USER' created."
else
  echo "User '$APP_USER' already exists."
fi

Using Lock Files to Prevent Concurrent Runs

Even an idempotent script can cause problems if two instances run simultaneously. A lock file ensures only one instance runs at a time.

Create a lock file at startup; remove it on exit.
Use a trap to clean up the lock even if the script is interrupted.
mkdir on a single path is atomic on most Linux filesystems — safer than touch for locking.

Without a lock, a slow cron job and a manual re-run can collide and corrupt shared state.

#!/usr/bin/env bash
set -euo pipefail

LOCKFILE="/tmp/myapp_deploy.lock"

# Atomic lock acquisition using mkdir
if ! mkdir "$LOCKFILE" 2>/dev/null; then
  echo "ERROR: Another instance is running (lock: $LOCKFILE). Exiting." >&2
  exit 1
fi

# Guarantee lock removal on any exit
trap 'rmdir "$LOCKFILE"; echo "Lock released."' EXIT

echo "Lock acquired. Running deployment..."
sleep 2   # simulate work
echo "Deployment complete."

Tracking Completed Steps with a State File

For multi-step scripts (migrations, installs, provisioning), you can track which steps have already completed using a state file. Each step checks the state file before executing and writes to it when done.

Cheap and portable — no database required.
Allows a failed script to resume from where it left off.
Store state in a predictable location like /var/lib/myapp/ or ~/.myapp/state/.

This pattern is used by major tools like apt, cloud-init, and database migration frameworks.

#!/usr/bin/env bash
set -euo pipefail

STATE_DIR="/tmp/myapp_state"
mkdir -p "$STATE_DIR"

run_step() {
  local step_name="$1"
  local step_cmd="$2"
  local marker="$STATE_DIR/${step_name}.done"

  if [ -f "$marker" ]; then
    echo "[SKIP] $step_name already completed."
    return 0
  fi

  echo "[RUN]  $step_name ..."
  eval "$step_cmd"
  touch "$marker"
  echo "[DONE] $step_name"
}

run_step "install_deps"   "echo 'Installing dependencies...'"
run_step "configure_db"   "echo 'Configuring database...'"
run_step "start_service"  "echo 'Starting service...'"

Introduction to Retry Logic

External calls — HTTP requests, DNS lookups, cloud API calls — are inherently unreliable. A single failure should not abort your entire script. Retry logic automatically re-attempts a failed command.

Naive retry: loop N times until the command succeeds.
Always set a maximum retry count to avoid infinite loops.
Log each attempt so failures are diagnosable.

The simplest retry function wraps any command and retries it up to a fixed number of times with a constant delay. This is good enough for many use cases but has a critical flaw under load — covered next.

#!/usr/bin/env bash
set -euo pipefail

# Simple fixed-delay retry (3 attempts, 2s apart)
retry() {
  local max_attempts=3
  local delay=2
  local attempt=1

  until "$@"; do
    if (( attempt >= max_attempts )); then
      echo "ERROR: Command failed after $max_attempts attempts: $*" >&2
      return 1
    fi
    echo "Attempt $attempt failed. Retrying in ${delay}s..." >&2
    sleep "$delay"
    (( attempt++ ))
  done
}

# Example: retry a curl call
retry curl --silent --fail --max-time 5 https://httpbin.org/get -o /dev/null
echo "Request succeeded."

The Thundering Herd Problem

When many clients retry simultaneously after a failure, they create a thundering herd — all retrying at the same fixed interval, hammering the server at the exact same moment and making recovery impossible.

100 scripts retry every 5 seconds → 100 simultaneous requests every 5 seconds.
The server is already struggling; the synchronized load makes it worse.
The solution is exponential backoff: double the wait time after each failure.
Add jitter (random noise) to desynchronize retries across clients.

Exponential backoff with jitter is the industry standard used by AWS SDKs, Google Cloud clients, and every major distributed system.

Implementing Exponential Backoff

Exponential backoff increases the wait time exponentially after each failure: 1s, 2s, 4s, 8s, 16s... This gives the remote system time to recover while reducing total load.

The formula: delay = base * (2 ^ attempt)

Set a cap (maximum delay) so waits don't grow unbounded.
Parameters to tune: base_delay, max_delay, max_attempts.

This function is reusable — pass any command to it as arguments.

#!/usr/bin/env bash
set -euo pipefail

retry_with_backoff() {
  local max_attempts="${RETRY_MAX_ATTEMPTS:-5}"
  local base_delay="${RETRY_BASE_DELAY:-1}"
  local max_delay="${RETRY_MAX_DELAY:-30}"
  local attempt=0
  local delay="$base_delay"

  until "$@"; do
    (( attempt++ ))
    if (( attempt >= max_attempts )); then
      echo "ERROR: '$*' failed after $max_attempts attempts." >&2
      return 1
    fi
    echo "Attempt $attempt failed. Backing off for ${delay}s..." >&2
    sleep "$delay"
    # Double the delay, but cap it
    delay=$(( delay * 2 ))
    (( delay > max_delay )) && delay=$max_delay
  done

  echo "Command succeeded on attempt $(( attempt + 1 ))."
}

retry_with_backoff echo "Simulated success"

Adding Jitter to Backoff

Jitter adds randomness to the backoff delay. Even with exponential backoff, if all clients start at the same time, they will still retry in sync. Jitter breaks this synchronization.

Two common jitter strategies:

Full jitter: sleep random(0, cap) — maximally spread, lowest peak load.
Equal jitter: sleep cap/2 + random(0, cap/2) — guarantees minimum wait, avoids hammering immediately.

In Bash, use $RANDOM (0–32767) to generate random numbers. Scale it to your delay range with modulo arithmetic.

#!/usr/bin/env bash
set -euo pipefail

retry_with_jitter() {
  local max_attempts="${1:-5}"; shift
  local base_delay=1
  local max_delay=32
  local attempt=0
  local cap=$base_delay

  until "$@"; do
    (( attempt++ ))
    if (( attempt >= max_attempts )); then
      echo "ERROR: Giving up after $max_attempts attempts." >&2
      return 1
    fi

    # Full jitter: random value in [0, cap]
    local jitter=$(( RANDOM % (cap + 1) ))
    echo "Attempt $attempt failed. Sleeping ${jitter}s (cap=${cap}s)..." >&2
    sleep "$jitter"

    # Grow cap exponentially, bounded by max_delay
    cap=$(( cap * 2 ))
    (( cap > max_delay )) && cap=$max_delay
  done
}

retry_with_jitter 4 curl --silent --fail --max-time 3 https://httpbin.org/get -o /dev/null
echo "Done."

Combining Idempotency and Retry in a Real Workflow

In production scripts, idempotency and retry logic work together. A typical deployment workflow might:

Acquire a lock (prevent concurrent runs)
Check state files (skip completed steps)
Use retry-with-backoff for external calls (download, API, DNS)
Mark steps done only after confirmed success
Release the lock via trap

This combination makes scripts safe to re-run at any point — after a crash, a timeout, or a manual abort — without leaving the system in a broken state.

#!/usr/bin/env bash
set -euo pipefail

STATE_DIR="/tmp/deploy_state" && mkdir -p "$STATE_DIR"
LOCK="/tmp/deploy.lock"

mkdir "$LOCK" 2>/dev/null || { echo "Already running."; exit 1; }
trap 'rmdir "$LOCK"' EXIT

step_done() { [ -f "$STATE_DIR/$1.done" ]; }
mark_done() { touch "$STATE_DIR/$1.done"; }

retry_backoff() {
  local attempt=0 delay=1
  until "${@:2}"; do
    (( ++attempt >= $1 )) && { echo "Failed after $1 attempts."; return 1; }
    echo "Retry $attempt in ${delay}s..."; sleep $delay; delay=$(( delay * 2 ))
  done
}

if ! step_done "download_artifact"; then
  retry_backoff 4 curl -fsSL https://httpbin.org/get -o /tmp/artifact.json
  mark_done "download_artifact"
  echo "[DONE] download_artifact"
else
  echo "[SKIP] download_artifact"
fi

echo "Deployment finished successfully."

Handling Non-Retriable Errors

Not all errors should be retried. Retrying a 404 Not Found or a 403 Forbidden is wasteful — they will never succeed without human intervention. Your retry logic should distinguish between:

Transient errors — network timeout, 503 Service Unavailable, DNS failure → retry
Permanent errors — 401 Unauthorized, 404 Not Found, invalid input → fail immediately

With curl, check the HTTP status code and skip retries for 4xx responses. Use --write-out '%{http_code}' to capture the status separately from the body.

#!/usr/bin/env bash
set -euo pipefail

fetch_with_retry() {
  local url="$1"
  local max_attempts=4
  local delay=1
  local attempt=0
  local http_code

  until http_code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 "$url"); do
    : # curl itself failed (network error)
    (( ++attempt >= max_attempts )) && { echo "Network error, giving up."; return 1; }
    sleep $delay; delay=$(( delay * 2 ))
  done

  # Permanent client errors — do not retry
  if [[ "$http_code" =~ ^4 ]]; then
    echo "ERROR: HTTP $http_code for $url — not retrying." >&2
    return 1
  fi

  # Transient server errors — retry
  if [[ "$http_code" =~ ^5 ]]; then
    (( ++attempt >= max_attempts )) && { echo "Server error, giving up."; return 1; }
    echo "HTTP $http_code — backing off ${delay}s..."
    sleep $delay; delay=$(( delay * 2 ))
    fetch_with_retry "$url"
    return
  fi

  echo "HTTP $http_code — success."
}

fetch_with_retry "https://httpbin.org/status/200"

Knowledge Check: Backoff Strategy Choice

A deployment script downloads a release artifact from an S3 bucket. During a recent incident, 80 pipeline instances all failed simultaneously due to a brief S3 outage. When S3 recovered after 10 seconds, all 80 instances retried at the same moment, causing another overload and extending the outage by 3 minutes.

Which retry strategy would best prevent this thundering-herd cascade in future incidents?

Recap: Idempotency and Retry-with-Backoff

In this lesson you learned how to design Bash scripts that are safe to re-run and resilient to transient failures.

Idempotency patterns:

Guard every operation with an existence check ([ -f ], [ -d ], id, getent).
Use mkdir -p and other built-in idempotent flags where available.
Use lock files (atomic mkdir) to prevent concurrent runs.
Use state files (marker files per step) to allow resume after failure.

Retry-with-backoff patterns:

Always set a maximum attempt count — never retry forever.
Use exponential backoff: double the delay after each failure.
Add jitter (randomness) to prevent thundering-herd synchronization.
Distinguish transient (retry) from permanent (fail fast) errors.

Combining these patterns produces scripts that are production-grade: safe, observable, and self-healing.

Frequently asked questions

Is the “Idempotent Scripts and Retry-with-Backoff Logic” lesson free?

Yes — the full text of “Idempotent Scripts and Retry-with-Backoff Logic” is free to read here on the web, and the DevOps Bootcamp course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the DevOps Bootcamp course, upgrade to CoddyKit PRO.

What will I learn in “Idempotent Scripts and Retry-with-Backoff Logic”?

Design operations that are safe to re-run and add exponential backoff for flaky external calls. You practise DevOps Bootcamp with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start DevOps Bootcamp?

No prior experience is required. DevOps Bootcamp on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Idempotent Scripts and Retry-with-Backoff Logic” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this DevOps Bootcamp lesson?

Yes. Every DevOps Bootcamp lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

Strict Mode with set -euo pipefail
Trap Handlers for Cleanup and Signals
Safe Temporary Files and Lock Directories
Idempotent Scripts and Retry-with-Backoff Logic

← Back to DevOps Bootcamp