0Pricing
DevOps Bootcamp · Lesson

Health Probes, Readiness Gates, and Wait Loops

Implement dependency wait loops and liveness probes that make containerized services start reliably.

Health Probes, Readiness Gates, and Wait Loops is a free DevOps Bootcamp lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the DevOps Bootcamp learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Services Fail at Startup

In containerized environments, services rarely start in isolation. A web application needs its database ready, a microservice needs its message broker available, and a worker needs its cache populated before it can process jobs.

Without coordination, containers start in parallel and the first requests arrive before dependencies are healthy. The result: connection refused errors, null pointer exceptions, and corrupted startup state that require manual restarts.

  • Liveness probe — is the process still running and not deadlocked?
  • Readiness probe — is the service ready to accept traffic?
  • Wait loop — a startup script that blocks until dependencies are reachable

Kubernetes provides built-in probe mechanisms, but the shell scripts that power initContainers, entrypoint.sh wrappers, and standalone health checks are written in Bash. Mastering them is a core DevOps skill.

The wait-for Pattern

The simplest dependency wait loop polls a target on a fixed interval until it becomes reachable. The canonical shape uses a while loop with nc (netcat) or curl to probe a TCP port or HTTP endpoint.

Key design decisions:

  • Timeout — bail out after N seconds so a broken dependency does not hang the pod forever
  • Backoff interval — sleep between probes to avoid hammering a recovering service
  • Exit code — exit 1 on timeout so the container restarts (or init container fails loudly)

The snippet below waits up to 60 seconds for a TCP port to accept connections before launching the main process.

#!/usr/bin/env bash
set -euo pipefail

HOST="${DB_HOST:-postgres}"
PORT="${DB_PORT:-5432}"
TIMEOUT=60
INTERVAL=2
ELAPSED=0

echo "[wait-for] Waiting for ${HOST}:${PORT}..."

until nc -z "$HOST" "$PORT" 2>/dev/null; do
  if (( ELAPSED >= TIMEOUT )); then
    echo "[wait-for] Timed out after ${TIMEOUT}s waiting for ${HOST}:${PORT}" >&2
    exit 1
  fi
  echo "[wait-for] ${HOST}:${PORT} not ready — retrying in ${INTERVAL}s (${ELAPSED}s elapsed)"
  sleep "$INTERVAL"
  (( ELAPSED += INTERVAL ))
done

echo "[wait-for] ${HOST}:${PORT} is up — continuing"
exec "$@"

HTTP Readiness Probes with curl

A TCP connection only tells you the port is open — not that the application behind it is ready to serve requests. Many services expose a dedicated /health or /readyz endpoint that returns HTTP 200 only when all internal subsystems are initialized.

Use curl --fail --silent --output /dev/null to probe an HTTP readiness endpoint. The --fail flag makes curl exit with code 22 on 4xx/5xx responses, which drives the retry logic cleanly.

Important flags to know:

  • --fail — treat HTTP errors as curl errors (non-zero exit)
  • --silent — suppress progress output
  • --max-time N — per-request timeout in seconds
  • --retry N --retry-delay S — curl's own retry layer (useful for simple cases)
#!/usr/bin/env bash
set -euo pipefail

HEALTH_URL="${HEALTH_URL:-http://localhost:8080/healthz}"
TIMEOUT=90
INTERVAL=3
ELAPSED=0

echo "[probe] Polling readiness at ${HEALTH_URL}"

until curl --fail --silent --output /dev/null \
           --max-time 2 "${HEALTH_URL}"; do
  if (( ELAPSED >= TIMEOUT )); then
    echo "[probe] Service not ready after ${TIMEOUT}s" >&2
    exit 1
  fi
  printf '[probe] Not ready yet (%ds elapsed)\n' "$ELAPSED"
  sleep "$INTERVAL"
  (( ELAPSED += INTERVAL ))
done

echo "[probe] Service is ready"
exec "$@"

Exponential Backoff in Wait Loops

A fixed-interval retry loop hammers a recovering service at a constant rate. Exponential backoff doubles the wait time on each attempt, reducing load during recovery while still converging quickly when the service comes up fast.

The standard formula is: sleep_time = min(base * 2^attempt, max_sleep). A jitter component (random fractional offset) prevents thundering herd problems when many containers restart simultaneously.

This pattern is used by production tools like wait-for-it, AWS SDK retries, and Kubernetes controller reconciliation loops.

#!/usr/bin/env bash
set -euo pipefail

HOST="${HOST:-redis}"
PORT="${PORT:-6379}"
MAX_ATTEMPTS=8
BASE_SLEEP=1
MAX_SLEEP=30

for attempt in $(seq 1 "$MAX_ATTEMPTS"); do
  if nc -z "$HOST" "$PORT" 2>/dev/null; then
    echo "[backoff] Connected to ${HOST}:${PORT} on attempt ${attempt}"
    exec "$@"
  fi

  # Exponential backoff with jitter
  raw=$(( BASE_SLEEP * (2 ** (attempt - 1)) ))
  capped=$(( raw < MAX_SLEEP ? raw : MAX_SLEEP ))
  jitter=$(( RANDOM % 3 ))
  sleep_time=$(( capped + jitter ))

  echo "[backoff] Attempt ${attempt}/${MAX_ATTEMPTS} failed — sleeping ${sleep_time}s"
  sleep "$sleep_time"
done

echo "[backoff] ${HOST}:${PORT} unreachable after ${MAX_ATTEMPTS} attempts" >&2
exit 1

Probing Multiple Dependencies

Real applications have multiple dependencies: a database, a cache, a message broker, and perhaps an external API. Probing them sequentially wastes startup time. A better approach probes all dependencies in parallel and waits for all to succeed.

The Bash & operator backgrounds each probe, and wait collects their exit codes. If any probe fails, the entrypoint exits non-zero, triggering a container restart.

Key technique: capture background PIDs with $! and pass them explicitly to wait so you can check individual exit codes.

#!/usr/bin/env bash
set -euo pipefail

wait_tcp() {
  local host="$1" port="$2" timeout="${3:-30}"
  local elapsed=0
  until nc -z "$host" "$port" 2>/dev/null; do
    (( elapsed >= timeout )) && { echo "TIMEOUT ${host}:${port}" >&2; return 1; }
    sleep 2; (( elapsed += 2 ))
  done
  echo "[ok] ${host}:${port}"
}

# Launch all probes in parallel
wait_tcp postgres 5432 60 &  PID_PG=$!
wait_tcp redis    6379 30 &  PID_RD=$!
wait_tcp rabbitmq 5672 45 &  PID_RQ=$!

# Collect results — fail fast if any probe failed
FAILED=0
for pid in $PID_PG $PID_RD $PID_RQ; do
  wait "$pid" || (( FAILED++ ))
done

if (( FAILED > 0 )); then
  echo "[entrypoint] ${FAILED} dependency probe(s) failed — aborting" >&2
  exit 1
fi

echo "[entrypoint] All dependencies ready"
exec "$@"

Liveness vs Readiness: Different Scripts for Different Probes

Kubernetes distinguishes between liveness and readiness probes, and they should do different things:

  • Liveness probe — answers: is the process still alive and not deadlocked? Should be fast and check only internal state (e.g., process PID file exists, a local health endpoint returns 200). A failing liveness probe kills and restarts the container.
  • Readiness probe — answers: should this pod receive traffic? May check downstream dependencies. A failing readiness probe removes the pod from the Service load balancer but does not restart it.

Never put slow dependency checks in liveness probes. A brief database outage would incorrectly restart all your application pods, compounding the problem.

#!/usr/bin/env bash
# liveness.sh — fast local-only check
# Used in: livenessProbe.exec.command
set -euo pipefail

PID_FILE="/var/run/app/app.pid"
HEALTH_URL="http://127.0.0.1:8080/internal/live"

# Check 1: process is running
[[ -f "$PID_FILE" ]] || { echo "PID file missing" >&2; exit 1; }
kill -0 "$(cat "$PID_FILE")" 2>/dev/null || { echo "Process dead" >&2; exit 1; }

# Check 2: local endpoint responds (2s timeout — never block)
curl --fail --silent --max-time 2 --output /dev/null "$HEALTH_URL" || {
  echo "Liveness endpoint unresponsive" >&2
  exit 1
}

echo "live"
exit 0

Startup Probes and initContainers

Kubernetes offers a third probe type: startupProbe. It runs instead of liveness/readiness probes until it succeeds once, giving slow-starting applications (JVM warm-up, database migrations) time to initialize without triggering false liveness failures.

For dependency waiting, initContainers are often cleaner than entrypoint scripts. They run before app containers start, and Kubernetes handles the retry/restart logic automatically. The init container image needs only sh, nc, or curl — you can use a minimal busybox or alpine image.

Example initContainer spec in a Pod manifest:

# kubernetes/pod-with-init.yaml (illustrative — not runnable as bash)
# initContainers run sequentially before app containers
initContainers:
  - name: wait-for-postgres
    image: busybox:1.36
    command:
      - sh
      - -c
      - |
        set -e
        echo 'Waiting for postgres...'
        until nc -z postgres 5432; do
          echo 'postgres not ready — sleeping 2s'
          sleep 2
        done
        echo 'postgres is up'

  - name: run-migrations
    image: myapp:latest
    command: ['python', 'manage.py', 'migrate', '--noinput']
    envFrom:
      - secretRef:
          name: app-secrets

Health Check Script with JSON Output

Production systems often aggregate health status across multiple subsystems and expose it as a structured JSON payload. This is consumed by load balancers, orchestrators, and monitoring dashboards.

A Bash health check script can build JSON output directly using printf or jq. The exit code still drives automation; the JSON body is for human operators and monitoring systems.

Convention: return HTTP 200 with {"status":"ok"} when healthy, HTTP 503 with {"status":"degraded", "checks":{...}} when unhealthy. The script below is meant to be served by a lightweight HTTP wrapper such as socat or called directly by Kubernetes exec probes.

#!/usr/bin/env bash
# health_check.sh — composite health with JSON output
set -uo pipefail

check_postgres() {
  pg_isready -h "${DB_HOST:-postgres}" -p "${DB_PORT:-5432}" \
             -U "${DB_USER:-app}" -t 2 &>/dev/null
}

check_redis() {
  redis-cli -h "${REDIS_HOST:-redis}" -p "${REDIS_PORT:-6379}" \
             PING 2>/dev/null | grep -q PONG
}

check_disk() {
  local usage
  usage=$(df / | awk 'NR==2{gsub(/%/,"",$5); print $5}')
  (( usage < 90 ))
}

PG_OK=0; RD_OK=0; DSK_OK=0
check_postgres && PG_OK=1
check_redis    && RD_OK=1
check_disk     && DSK_OK=1

OVERALL=$(( PG_OK && RD_OK && DSK_OK ))
STATUS=$( (( OVERALL )) && echo 'ok' || echo 'degraded' )

printf '{"status":"%s","checks":{"postgres":%s,"redis":%s,"disk":%s}}\n' \
  "$STATUS" "$PG_OK" "$RD_OK" "$DSK_OK"

(( OVERALL )) && exit 0 || exit 1

Timeout Utility with the deadline Pattern

The GNU timeout command wraps any command and kills it if it does not finish within the specified duration. It is the cleanest way to enforce a hard deadline on a wait loop or health probe without managing background jobs manually.

timeout DURATION COMMAND [ARGS...]

Exit codes from timeout:

  • 0 — command succeeded within the deadline
  • Exit code of the command — command ran but returned non-zero
  • 124 — command timed out (SIGTERM sent)
  • 137 — command killed with SIGKILL (after --kill-after)

Detecting the 124 exit code lets you print a clear timeout message instead of a generic error.

#!/usr/bin/env bash
set -euo pipefail

HOST="${DB_HOST:-postgres}"
PORT="${DB_PORT:-5432}"
DEADLINE=60  # seconds

# Inline poll loop, wrapped by timeout
timeout "$DEADLINE" bash -c "
  until nc -z '$HOST' '$PORT' 2>/dev/null; do
    echo '[wait] ${HOST}:${PORT} not ready...'
    sleep 2
  done
" && echo "[wait] ${HOST}:${PORT} is up" || {
  code=$?
  if (( code == 124 )); then
    echo "[wait] Timed out after ${DEADLINE}s waiting for ${HOST}:${PORT}" >&2
  else
    echo "[wait] Probe failed with exit code ${code}" >&2
  fi
  exit "$code"
}

exec "$@"

Self-contained wait-for-it in Pure Bash

Many minimal container images lack nc (netcat). Pure Bash can open TCP connections using process substitution to /dev/tcp, a non-standard but widely supported Bash built-in that requires zero external tools.

Syntax: /dev/tcp/HOST/PORT — Bash opens a TCP connection when you redirect to or from this path. It raises an error (non-zero exit) if the connection is refused or times out.

This is the technique used by the popular wait-for-it.sh script that ships with many Docker Compose setups. The script below is a complete, standalone implementation you can COPY into any Dockerfile.

#!/usr/bin/env bash
# wait-for-it.sh (pure bash, no nc/curl required)
set -uo pipefail

usage() { echo "Usage: $0 HOST:PORT [-t TIMEOUT] [-- COMMAND]"; exit 1; }

parse_hostport() {
  HOST="${1%%:*}"
  PORT="${1##*:}"
  [[ -n "$HOST" && "$PORT" =~ ^[0-9]+$ ]] || usage
}

[[ $# -ge 1 ]] || usage
parse_hostport "$1"; shift

TIMEOUT=30
[[ "${1:-}" == "-t" ]] && { TIMEOUT="$2"; shift 2; }
[[ "${1:-}" == "--" ]] && shift

wait_for() {
  local elapsed=0
  while (( elapsed < TIMEOUT )); do
    # Pure bash TCP probe — no nc, no curl
    if (exec 3<>"/dev/tcp/${HOST}/${PORT}") 2>/dev/null; then
      exec 3>&-
      return 0
    fi
    sleep 1
    (( elapsed++ ))
  done
  return 1
}

echo "Waiting for ${HOST}:${PORT} (timeout ${TIMEOUT}s)..."
if wait_for; then
  echo "${HOST}:${PORT} is available"
  [[ $# -gt 0 ]] && exec "$@"
else
  echo "Timed out waiting for ${HOST}:${PORT}" >&2
  exit 1
fi

Integrating Probes into entrypoint.sh

The entrypoint pattern is the standard way to combine wait loops, environment validation, and process startup in a single maintainable script. Docker's ENTRYPOINT calls this script, and the script ends with exec "$@" to hand off to the CMD with the same PID (enabling clean signal forwarding).

A production-grade entrypoint.sh typically:

  • Validates required environment variables early (fail fast)
  • Runs dependency wait loops
  • Executes database migrations (if applicable)
  • Runs a final self-check
  • Transfers control with exec "$@"

Using exec is critical — it replaces the shell process so the app becomes PID 1 and receives SIGTERM from Docker/Kubernetes graceful shutdown directly.

#!/usr/bin/env bash
# docker/entrypoint.sh
set -euo pipefail

# ── 1. Validate required env vars ────────────────────────────────────
for var in DATABASE_URL REDIS_URL SECRET_KEY; do
  [[ -n "${!var:-}" ]] || { echo "FATAL: ${var} is not set" >&2; exit 1; }
done

# ── 2. Parse DB host/port from DATABASE_URL ──────────────────────────
DB_HOST=$(echo "$DATABASE_URL" | sed -E 's|.*@([^:/]+).*|\1|')
DB_PORT=$(echo "$DATABASE_URL" | sed -E 's|.*:([0-9]+)/.*|\1|')

# ── 3. Wait for dependencies ─────────────────────────────────────────
timeout 60 bash -c "
  until nc -z '${DB_HOST}' '${DB_PORT}' 2>/dev/null; do sleep 2; done
" || { echo "Database unreachable" >&2; exit 1; }

# ── 4. Run migrations ────────────────────────────────────────────────
echo "[entrypoint] Running migrations..."
python manage.py migrate --noinput

# ── 5. Hand off to CMD (exec preserves PID 1 for signal handling) ────
echo "[entrypoint] Starting application: $*"
exec "$@"

Knowledge Check: Liveness vs Readiness Probes

Test your understanding of the key concepts covered in this lesson.

Recap: Health Probes, Readiness Gates, and Wait Loops

In this lesson you built the Bash toolkit for reliable container startup coordination. Here is what you covered:

  • Wait-for pattern — a until nc -z HOST PORT loop with a hard timeout and elapsed-time guard prevents infinite blocking on broken dependencies.
  • HTTP readiness probes — curl --fail --max-time validates that an application is truly ready, not just that the port is open.
  • Exponential backoff — doubling the sleep interval between retries reduces thundering herd load and lets recovering services stabilize.
  • Parallel dependency probing — backgrounding probes with & and collecting results with wait $PID cuts startup latency when multiple dependencies exist.
  • Liveness vs Readiness — liveness probes must be fast and local-only; readiness probes may check downstream systems. Never mix them up.
  • startupProbe — protects slow-starting apps from false liveness failures during initialization.
  • /dev/tcp probe — pure Bash TCP check requires no external tools, ideal for minimal container images.
  • entrypoint.sh — env validation, wait loops, migrations, and exec "$@" form the standard containerized service startup pattern.

These patterns form the foundation of self-healing, production-grade container deployments across Docker Compose, Kubernetes, and ECS.

Frequently asked questions

Is the “Health Probes, Readiness Gates, and Wait Loops” lesson free?

Yes — the full text of “Health Probes, Readiness Gates, and Wait Loops” is free to read here on the web, and the DevOps Bootcamp course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the DevOps Bootcamp course, upgrade to CoddyKit PRO.

What will I learn in “Health Probes, Readiness Gates, and Wait Loops”?

Implement dependency wait loops and liveness probes that make containerized services start reliably. You practise DevOps Bootcamp with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start DevOps Bootcamp?

No prior experience is required. DevOps Bootcamp on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Health Probes, Readiness Gates, and Wait Loops” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this DevOps Bootcamp lesson?

Yes. Every DevOps Bootcamp lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Writing Lean Dockerfiles and Shell Entrypoints
  2. Templating Configs with envsubst and heredocs
  3. Scripting Cloud Resources via CLI and jq
  4. Health Probes, Readiness Gates, and Wait Loops
← Back to DevOps Bootcamp