DevOps Bootcamp · Lesson

Scripting Cloud Resources via CLI and jq

Drive cloud provider CLIs idempotently and parse JSON responses to provision and tear down resources.

Scripting Cloud Resources via CLI and jq is a free DevOps Bootcamp lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the DevOps Bootcamp learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Idempotent Cloud Scripting Matters

At C2 level, cloud scripting is not about clicking buttons — it is about writing code that can run safely multiple times without creating duplicate resources or failing on the second run.

An idempotent script checks whether a resource already exists before creating it. This is the cornerstone of reliable infrastructure automation.

Cloud CLIs (AWS, GCP, Azure) return JSON — parsing that output is essential.
jq is the standard Unix tool for slicing, filtering, and transforming JSON from shell scripts.
Combining CLI + jq + conditional logic lets you write robust, repeatable provisioning scripts.

Throughout this lesson you will provision S3 buckets, EC2 instances, and IAM roles using AWS CLI as the reference, with patterns that transfer directly to gcloud and az.

Installing and Verifying Cloud CLIs

Before scripting, confirm the right tools are present. Always version-pin in CI to avoid drift between environments.

The snippet below checks for AWS CLI v2, jq, and the GCP SDK, installing only what is missing — a pattern useful in bootstrap scripts for fresh VMs or containers.

#!/usr/bin/env bash
set -euo pipefail

check_or_install() {
  local cmd="$1"
  local install_cmd="$2"
  if ! command -v "$cmd" &>/dev/null; then
    echo "[INFO] $cmd not found — installing..."
    eval "$install_cmd"
  else
    echo "[OK]   $cmd $("$cmd" --version 2>&1 | head -1)"
  fi
}

# AWS CLI v2
check_or_install aws \
  'curl -fsSL https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o /tmp/awscliv2.zip && unzip -q /tmp/awscliv2.zip -d /tmp && sudo /tmp/aws/install'

# jq
check_or_install jq \
  'sudo apt-get install -y jq 2>/dev/null || sudo yum install -y jq'

# gcloud (optional)
check_or_install gcloud \
  'echo "Install gcloud SDK manually from https://cloud.google.com/sdk"'

echo "All prerequisites satisfied."

Querying Existing Resources with jq

The first step of any idempotent script is a read — ask the API whether the resource already exists, then branch accordingly.

AWS CLI always returns JSON. jq lets you extract exactly the field you need:

jq -r '.Buckets[].Name' — raw string output, one bucket name per line.
jq -e — exits with code 1 if the expression produces null or false, making it ideal for if guards.
jq '.[] | select(.Name == env.BUCKET)' — filter using a shell variable via env.

The snippet below lists all S3 buckets and checks whether a target bucket already exists.

#!/usr/bin/env bash
set -euo pipefail

BUCKET="my-devops-artifacts-$(date +%Y%m)"

echo "Fetching existing S3 buckets..."
EXISTING=$(aws s3api list-buckets --output json)

# Extract names as newline-separated list
echo "$EXISTING" | jq -r '.Buckets[].Name'

# Check if target bucket exists
if echo "$EXISTING" | jq -e --arg b "$BUCKET" '.Buckets[] | select(.Name == $b)' > /dev/null 2>&1; then
  echo "[EXISTS] Bucket $BUCKET already present — skipping creation."
else
  echo "[MISSING] Bucket $BUCKET not found — will create."
fi

Idempotent S3 Bucket Creation

With the existence check in place, wrap the creation in a guard. A well-structured cloud function follows this pattern:

Read current state from the API.
Compare desired state to actual state.
Act only on the diff.

Note the --create-bucket-configuration flag — it is required for all regions except us-east-1. Hardcoding the region inside the script avoids silent failures when AWS_DEFAULT_REGION is unset.

#!/usr/bin/env bash
set -euo pipefail

REGION="eu-west-1"
BUCKET="my-devops-artifacts-$(date +%Y%m)"

ensure_bucket() {
  local bucket="$1"
  local region="$2"

  local existing
  existing=$(aws s3api list-buckets --query 'Buckets[].Name' --output json)

  if echo "$existing" | jq -e --arg b "$bucket" 'index($b) != null' > /dev/null 2>&1; then
    echo "[SKIP] Bucket $bucket already exists."
    return 0
  fi

  echo "[CREATE] Creating bucket $bucket in $region..."
  aws s3api create-bucket \
    --bucket "$bucket" \
    --region "$region" \
    --create-bucket-configuration LocationConstraint="$region"

  # Enable versioning immediately after creation
  aws s3api put-bucket-versioning \
    --bucket "$bucket" \
    --versioning-configuration Status=Enabled

  echo "[DONE] Bucket $bucket created with versioning enabled."
}

ensure_bucket "$BUCKET" "$REGION"

Parsing Nested JSON: EC2 Instance State

EC2 responses are deeply nested. jq path traversal and --query (JMESPath, native to AWS CLI) both work — but jq is more powerful for complex logic.

Key jq patterns for EC2:

.Reservations[].Instances[] — flatten the double-array structure.
select(.State.Name == "running") — filter by state.
.Tags[] | select(.Key == "Name") | .Value — extract a tag value.

The snippet finds a running instance by its Name tag and returns its ID and private IP.

#!/usr/bin/env bash
set -euo pipefail

INSTANCE_NAME="web-server-prod"

RESULT=$(aws ec2 describe-instances \
  --filters \
    "Name=tag:Name,Values=${INSTANCE_NAME}" \
    "Name=instance-state-name,Values=running" \
  --output json)

# Extract instance ID and private IP using jq
INSTANCE_ID=$(echo "$RESULT" | jq -r \
  '.Reservations[].Instances[] | .InstanceId')

PRIVATE_IP=$(echo "$RESULT" | jq -r \
  '.Reservations[].Instances[] | .PrivateIpAddress')

if [[ -z "$INSTANCE_ID" ]]; then
  echo "[WARN] No running instance named '$INSTANCE_NAME' found."
  exit 1
fi

echo "Instance ID : $INSTANCE_ID"
echo "Private IP  : $PRIVATE_IP"

Idempotent IAM Role Provisioning

IAM resources are global and must not be duplicated. AWS returns a specific error code — EntityAlreadyExists — when you try to create a role that already exists. Catching that code is a cleaner idempotency pattern than a pre-check list call when dealing with IAM at scale.

The script below demonstrates:

Capturing the CLI exit code with || true to prevent set -e from aborting.
Parsing the error message JSON that AWS writes to stderr using process substitution.
Attaching a policy only if it is not already attached.

#!/usr/bin/env bash
set -euo pipefail

ROLE_NAME="DevOpsDeployRole"
POLICY_ARN="arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess"

TRUST_POLICY='{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Service": "ec2.amazonaws.com" },
    "Action": "sts:AssumeRole"
  }]
}'

# Attempt creation; ignore EntityAlreadyExists
CREATE_OUTPUT=$(aws iam create-role \
  --role-name "$ROLE_NAME" \
  --assume-role-policy-document "$TRUST_POLICY" \
  --output json 2>&1) || {
  if echo "$CREATE_OUTPUT" | grep -q 'EntityAlreadyExists'; then
    echo "[SKIP] Role $ROLE_NAME already exists."
  else
    echo "[ERROR] Unexpected error: $CREATE_OUTPUT" >&2
    exit 1
  fi
}

# Attach policy (attach-role-policy is idempotent by default)
aws iam attach-role-policy \
  --role-name "$ROLE_NAME" \
  --policy-arn "$POLICY_ARN"

echo "[OK] Role $ROLE_NAME ready with policy $POLICY_ARN."

jq Advanced: Transforms, Maps, and toentries

Real infrastructure responses contain dozens of fields. jq transforms let you reshape output for downstream tools, logs, or config files.

Essential advanced patterns:

map(select(...)) — filter an array without losing the array wrapper.
to_entries | map(select(.value != null)) — remove null fields before writing to a config.
[.[] | {id: .InstanceId, ip: .PrivateIpAddress}] — project to a new shape.
@csv, @tsv, @base64 — built-in format converters.

The snippet below extracts all running instances and writes a TSV inventory file.

#!/usr/bin/env bash
set -euo pipefail

OUTPUT_FILE="/tmp/ec2_inventory.tsv"

aws ec2 describe-instances \
  --filters "Name=instance-state-name,Values=running" \
  --output json \
| jq -r '
  ["InstanceId", "Name", "PrivateIp", "Type", "AZ"],
  [
    .Reservations[].Instances[] | [
      .InstanceId,
      (.Tags // [] | map(select(.Key == "Name")) | .[0].Value // "(none)"),
      (.PrivateIpAddress // "N/A"),
      .InstanceType,
      .Placement.AvailabilityZone
    ]
  ][]
| @tsv' > "$OUTPUT_FILE"

echo "Inventory written to $OUTPUT_FILE:"
column -t "$OUTPUT_FILE"

Waiting for Async Operations: Polling with jq

Cloud operations are asynchronous. Creating an EC2 instance returns immediately with state pending. Reliable scripts must poll until the desired state is reached before continuing.

The pattern below uses a until loop with exponential back-off. AWS CLI also provides wait subcommands (e.g. aws ec2 wait instance-running), but hand-rolled polling gives you custom timeout control and richer logging.

#!/usr/bin/env bash
set -euo pipefail

INSTANCE_ID="i-0abcdef1234567890"
MAX_WAIT=300   # seconds
INTERVAL=10
ELAPSED=0

echo "Waiting for instance $INSTANCE_ID to reach 'running' state..."

while true; do
  STATE=$(aws ec2 describe-instances \
    --instance-ids "$INSTANCE_ID" \
    --output json \
  | jq -r '.Reservations[0].Instances[0].State.Name')

  echo "  [$(date +%T)] state = $STATE"

  [[ "$STATE" == "running" ]] && break

  if [[ "$STATE" == "terminated" || "$STATE" == "shutting-down" ]]; then
    echo "[FATAL] Instance entered terminal state: $STATE" >&2
    exit 1
  fi

  if (( ELAPSED >= MAX_WAIT )); then
    echo "[TIMEOUT] Instance did not reach 'running' after ${MAX_WAIT}s." >&2
    exit 1
  fi

  sleep "$INTERVAL"
  (( ELAPSED += INTERVAL ))
done

echo "[OK] Instance $INSTANCE_ID is running."

Multi-Cloud Pattern: GCP and Azure Equivalents

The idempotent read-then-act pattern transfers directly to other cloud CLIs. Both gcloud and az return JSON and support filtering:

GCP: gcloud ... --format='json' — pipe to jq exactly as with AWS. Use gcloud ... --quiet to suppress prompts in scripts.
Azure: az ... --output json — identical pattern. az group exists returns a plain boolean string (true/false), skipping the need for jq in simple cases.

The snippet shows idempotent resource group creation in Azure and GCS bucket creation in GCP side-by-side using the same guard pattern.

#!/usr/bin/env bash
set -euo pipefail

# --- Azure: idempotent resource group ---
RG="devops-rg"
LOCATION="westeurope"

if [[ $(az group exists --name "$RG") == "true" ]]; then
  echo "[SKIP] Azure resource group $RG already exists."
else
  echo "[CREATE] Creating Azure resource group $RG..."
  az group create --name "$RG" --location "$LOCATION" --output json \
    | jq '{name: .name, location: .location, provisioningState: .properties.provisioningState}'
fi

# --- GCP: idempotent GCS bucket ---
GCS_BUCKET="gs://devops-artifacts-prod"
PROJECT="my-gcp-project"

if gcloud storage buckets describe "$GCS_BUCKET" \
     --project="$PROJECT" --format='value(name)' &>/dev/null; then
  echo "[SKIP] GCS bucket $GCS_BUCKET already exists."
else
  echo "[CREATE] Creating GCS bucket $GCS_BUCKET..."
  gcloud storage buckets create "$GCS_BUCKET" \
    --project="$PROJECT" \
    --location=EU \
    --uniform-bucket-level-access
fi

Teardown: Safe Resource Destruction

Destruction scripts are as important as creation scripts. A safe teardown:

Lists resources before deleting anything and prints a summary for human review.
Accepts a --dry-run flag so operators can confirm the plan without acting.
Deletes in the correct dependency order (e.g. terminate instances before deleting security groups).

The snippet below terminates all EC2 instances tagged Env=staging with a dry-run guard.

#!/usr/bin/env bash
set -euo pipefail

DRY_RUN="${1:-}"

echo "Finding staging EC2 instances..."

INSTANCE_IDS=$(aws ec2 describe-instances \
  --filters \
    "Name=tag:Env,Values=staging" \
    "Name=instance-state-name,Values=running,stopped" \
  --output json \
| jq -r '[.Reservations[].Instances[].InstanceId] | @sh')

if [[ -z "$INSTANCE_IDS" ]]; then
  echo "[INFO] No staging instances found. Nothing to do."
  exit 0
fi

echo "Instances to terminate: $INSTANCE_IDS"

if [[ "$DRY_RUN" == "--dry-run" ]]; then
  echo "[DRY-RUN] No changes made."
  exit 0
fi

read -rp "Terminate these instances? [yes/N]: " CONFIRM
[[ "$CONFIRM" != "yes" ]] && { echo "Aborted."; exit 0; }

# shellcheck disable=SC2086
aws ec2 terminate-instances --instance-ids $INSTANCE_IDS --output json \
| jq '.TerminatingInstances[] | {id: .InstanceId, state: .CurrentState.Name}'

echo "[DONE] Termination initiated."

End-to-End: Idempotent Infrastructure Bootstrap Script

Bringing it all together: a production-grade bootstrap script orchestrates multiple resources in the correct order, is fully idempotent, and emits structured logs that a CI system can parse.

Key practices demonstrated:

Structured logging via a log() helper that prefixes [INFO], [WARN], [ERROR].
State file — write created resource IDs to a JSON state file so subsequent runs and teardown scripts share the same references.
Error trap — trap catches unexpected exits and reports the failing line number.

#!/usr/bin/env bash
set -euo pipefail

STATE_FILE="/tmp/infra_state.json"
REGION="eu-west-1"
BUCKET="devops-bootstrap-$(date +%Y%m)"
ROLE="BootstrapRole"

log() { echo "[$(date -u +%T)] [$1] ${*:2}"; }
trap 'log ERROR "Script failed at line $LINENO"' ERR

# Initialize state
[[ -f "$STATE_FILE" ]] || echo '{}' > "$STATE_FILE"

# --- Step 1: S3 bucket ---
EXISTING_BUCKETS=$(aws s3api list-buckets --query 'Buckets[].Name' --output json)
if echo "$EXISTING_BUCKETS" | jq -e --arg b "$BUCKET" 'index($b) != null' > /dev/null; then
  log INFO "Bucket $BUCKET exists — skipping."
else
  aws s3api create-bucket --bucket "$BUCKET" --region "$REGION" \
    --create-bucket-configuration LocationConstraint="$REGION" > /dev/null
  log INFO "Bucket $BUCKET created."
fi

# Update state file
jq --arg b "$BUCKET" '.bucket = $b' "$STATE_FILE" > /tmp/_state_tmp && mv /tmp/_state_tmp "$STATE_FILE"

# --- Step 2: IAM role ---
if aws iam get-role --role-name "$ROLE" &>/dev/null; then
  log INFO "Role $ROLE exists — skipping."
else
  aws iam create-role --role-name "$ROLE" \
    --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}]}' \
    --output json | jq '{RoleName: .Role.RoleName, Arn: .Role.Arn}'
  log INFO "Role $ROLE created."
fi

jq --arg r "$ROLE" '.role = $r' "$STATE_FILE" > /tmp/_state_tmp && mv /tmp/_state_tmp "$STATE_FILE"

log INFO "Bootstrap complete. State: $(cat "$STATE_FILE" | jq -c .)"

Knowledge Check: jq Idempotency Guard

Test your understanding of idempotent cloud scripting with jq.

Recap: Scripting Cloud Resources via CLI and jq

This lesson covered the full lifecycle of idempotent cloud automation using shell scripts, cloud CLIs, and jq.

Core principles:

Read before write — always query existing state first; only act on the diff.
jq -e for guards — use exit-status mode to drive if branches from JSON responses.
Error code semantics — catch provider-specific error codes (e.g. EntityAlreadyExists) instead of pre-listing when that is more efficient.
Poll for async state — use until loops with timeouts; never assume a resource is ready immediately after creation.
State files — write resource IDs to a shared JSON file so each phase of the script and teardown share the same references.
Dry-run flags — always support --dry-run for safe operator review before destructive actions.

These patterns — combined with a strict set -euo pipefail header and an ERR trap — form the foundation of production-grade infrastructure scripting at C2 level.

Frequently asked questions

Is the “Scripting Cloud Resources via CLI and jq” lesson free?

Yes — the full text of “Scripting Cloud Resources via CLI and jq” is free to read here on the web, and the DevOps Bootcamp course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the DevOps Bootcamp course, upgrade to CoddyKit PRO.

What will I learn in “Scripting Cloud Resources via CLI and jq”?

Drive cloud provider CLIs idempotently and parse JSON responses to provision and tear down resources. You practise DevOps Bootcamp with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start DevOps Bootcamp?

No prior experience is required. DevOps Bootcamp on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Scripting Cloud Resources via CLI and jq” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this DevOps Bootcamp lesson?

Yes. Every DevOps Bootcamp lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

Writing Lean Dockerfiles and Shell Entrypoints
Templating Configs with envsubst and heredocs
Scripting Cloud Resources via CLI and jq
Health Probes, Readiness Gates, and Wait Loops

← Back to DevOps Bootcamp