DevOps Bootcamp · Lesson

Advanced Scripting Project

Work through a comprehensive project that integrates all learned concepts, from command-line tools to advanced Bash scripting techniques.

Advanced Scripting Project is a free DevOps Bootcamp lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the DevOps Bootcamp learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Project Start: The Challenge

Welcome to our final project! Throughout this course, you've learned many powerful Linux commands and Bash scripting techniques. Now, it's time to bring them all together.

In this lesson, we'll build a practical, advanced Bash script from scratch. This project will challenge you to integrate everything you've learned into a single, functional solution.

Define Our Project Goal

Our goal is to create a file monitoring and archiving script. Imagine you have a server that generates log files, and you need to process them regularly without manual intervention.

Here's what our script will do:

Continuously monitor a specific directory for new text files.
Process each new file (e.g., count its lines).
Move the processed file to an archive directory.
Log all activities, including timestamps and any errors.

Script Foundation: Shebang & Variables

Every good script starts with a clear foundation. We'll begin by defining our shebang and setting up key variables. These variables will make our script flexible and easy to configure.

We'll define paths for our monitoring directory, archive directory, and a log file.

#!/bin/bash

# --- Configuration Variables ---
MONITOR_DIR="/tmp/coddykit_monitor"
ARCHIVE_DIR="/tmp/coddykit_archive"
LOG_FILE="/tmp/coddykit_script.log"

# Log script start
echo "$(date): Script initialized." >> "$LOG_FILE"

Ensuring Directories Exist

Before our script can monitor or archive, we need to ensure the necessary directories exist. Using mkdir -p is crucial here, as it creates directories only if they don't already exist, and it won't throw an error if they do.

We'll also add basic error handling to exit gracefully if directory creation fails.

#!/bin/bash

# --- Configuration Variables ---
MONITOR_DIR="/tmp/coddykit_monitor"
ARCHIVE_DIR="/tmp/coddykit_archive"
LOG_FILE="/tmp/coddykit_script.log"

# Log script start
echo "$(date): Script initialized." >> "$LOG_FILE"

# Ensure monitoring and archive directories exist
mkdir -p "$MONITOR_DIR" || { echo "$(date): ERROR: Cannot create $MONITOR_DIR. Exiting." >> "$LOG_FILE"; exit 1; }
mkdir -p "$ARCHIVE_DIR" || { echo "$(date): ERROR: Cannot create $ARCHIVE_DIR. Exiting." >> "$LOG_FILE"; exit 1; }

echo "$(date): Directories checked/created." >> "$LOG_FILE"

The Monitoring Loop

Our script needs to run continuously to monitor for new files. A while true loop is perfect for this. Inside the loop, we'll use sleep to pause for a few seconds between checks, preventing it from consuming too many resources.

We'll also log each check to keep track of activity.

#!/bin/bash

# --- Configuration Variables ---
MONITOR_DIR="/tmp/coddykit_monitor"
ARCHIVE_DIR="/tmp/coddykit_archive"
LOG_FILE="/tmp/coddykit_script.log"

# Log script start
echo "$(date): Script initialized." >> "$LOG_FILE"

# Ensure monitoring and archive directories exist
mkdir -p "$MONITOR_DIR" || { echo "$(date): ERROR: Cannot create $MONITOR_DIR. Exiting." >> "$LOG_FILE"; exit 1; }
mkdir -p "$ARCHIVE_DIR" || { echo "$(date): ERROR: Cannot create $ARCHIVE_DIR. Exiting." >> "$LOG_FILE"; exit 1; }

echo "$(date): Directories checked/created." >> "$LOG_FILE"

# --- Main Monitoring Loop ---
while true; do
  echo "$(date): Checking for new files in $MONITOR_DIR..." >> "$LOG_FILE"
  # File processing will go here
  sleep 5 # Check every 5 seconds
done

Finding & Processing Files

Inside our loop, we need to find new text files. The find command is ideal for this, combined with a for loop to process each discovered file. We'll use wc -l to count the lines in each file as our processing step.

This demonstrates combining command-line tools within a script.

#!/bin/bash

# --- Configuration Variables ---
MONITOR_DIR="/tmp/coddykit_monitor"
ARCHIVE_DIR="/tmp/coddykit_archive"
LOG_FILE="/tmp/coddykit_script.log"

# Log script start
echo "$(date): Script initialized." >> "$LOG_FILE"

# Ensure monitoring and archive directories exist
mkdir -p "$MONITOR_DIR" || { echo "$(date): ERROR: Cannot create $MONITOR_DIR. Exiting." >> "$LOG_FILE"; exit 1; }
mkdir -p "$ARCHIVE_DIR" || { echo "$(date): ERROR: Cannot create $ARCHIVE_DIR. Exiting." >> "$LOG_FILE"; exit 1; }

echo "$(date): Directories checked/created." >> "$LOG_FILE"

# --- Main Monitoring Loop ---
while true; do
  echo "$(date): Checking for new files in $MONITOR_DIR..." >> "$LOG_FILE"
  # Find new .txt files and process them
  for file in $(find "$MONITOR_DIR" -maxdepth 1 -type f -name "*.txt"); do
    if [ -f "$file" ]; then # Double-check if it's a regular file
      LINE_COUNT=$(wc -l < "$file")
      echo "$(date): Found '$file'. Lines: $LINE_COUNT." >> "$LOG_FILE"
      # Archiving will go here
    fi
  done
  sleep 5 # Check every 5 seconds
done

Archiving Processed Files

After processing a file, we need to move it to our archive directory. The mv command is used for this. We'll also add an if statement to check the exit status of mv, ensuring the move was successful and logging any failures.

This adds robustness to our script.

#!/bin/bash

# --- Configuration Variables ---
MONITOR_DIR="/tmp/coddykit_monitor"
ARCHIVE_DIR="/tmp/coddykit_archive"
LOG_FILE="/tmp/coddykit_script.log"

# Log script start
echo "$(date): Script initialized." >> "$LOG_FILE"

# Ensure monitoring and archive directories exist
mkdir -p "$MONITOR_DIR" || { echo "$(date): ERROR: Cannot create $MONITOR_DIR. Exiting." >> "$LOG_FILE"; exit 1; }
mkdir -p "$ARCHIVE_DIR" || { echo "$(date): ERROR: Cannot create $ARCHIVE_DIR. Exiting." >> "$LOG_FILE"; exit 1; }

echo "$(date): Directories checked/created." >> "$LOG_FILE"

# --- Main Monitoring Loop ---
while true; do
  echo "$(date): Checking for new files in $MONITOR_DIR..." >> "$LOG_FILE"
  for file in $(find "$MONITOR_DIR" -maxdepth 1 -type f -name "*.txt"); do
    if [ -f "$file" ]; then
      LINE_COUNT=$(wc -l < "$file")
      echo "$(date): Found '$file'. Lines: $LINE_COUNT." >> "$LOG_FILE"
      
      mv "$file" "$ARCHIVE_DIR/"
      if [ $? -eq 0 ]; then
        echo "$(date): Moved '$file' to '$ARCHIVE_DIR'." >> "$LOG_FILE"
      else
        echo "$(date): ERROR: Failed to move '$file' to '$ARCHIVE_DIR'." >> "$LOG_FILE"
      fi
    fi
  done
  sleep 5
done

Adding Graceful Exit with Trap

A long-running script needs a way to stop gracefully. The trap command allows us to catch signals, like SIGINT (triggered by Ctrl+C), and execute a cleanup function before exiting. This ensures our script logs its shutdown.

This is a crucial step for production-ready scripts.

#!/bin/bash

# --- Configuration Variables ---
MONITOR_DIR="/tmp/coddykit_monitor"
ARCHIVE_DIR="/tmp/coddykit_archive"
LOG_FILE="/tmp/coddykit_script.log"

# --- Functions ---
cleanup() {
  echo "$(date): Script received stop signal. Exiting gracefully." >> "$LOG_FILE"
  exit 0 # Exit with success status
}

# --- Trap Signals ---
# Trap SIGINT (Ctrl+C) and call the cleanup function
trap cleanup SIGINT

# Log script start
echo "$(date): Script initialized." >> "$LOG_FILE"

# Ensure monitoring and archive directories exist
mkdir -p "$MONITOR_DIR" || { echo "$(date): ERROR: Cannot create $MONITOR_DIR. Exiting." >> "$LOG_FILE"; exit 1; }
mkdir -p "$ARCHIVE_DIR" || { echo "$(date): ERROR: Cannot create $ARCHIVE_DIR. Exiting." >> "$LOG_FILE"; exit 1; }

echo "$(date): Directories checked/created." >> "$LOG_FILE"

# --- Main Monitoring Loop ---
while true; do
  echo "$(date): Checking for new files in $MONITOR_DIR..." >> "$LOG_FILE"
  for file in $(find "$MONITOR_DIR" -maxdepth 1 -type f -name "*.txt"); do
    if [ -f "$file" ]; then
      LINE_COUNT=$(wc -l < "$file")
      echo "$(date): Found '$file'. Lines: $LINE_COUNT." >> "$LOG_FILE"
      
      mv "$file" "$ARCHIVE_DIR/"
      if [ $? -eq 0 ]; then
        echo "$(date): Moved '$file' to '$ARCHIVE_DIR'." >> "$LOG_FILE"
      else
        echo "$(date): ERROR: Failed to move '$file' to '$ARCHIVE_DIR'." >> "$LOG_FILE"
      fi
    fi
  done
  sleep 5
done

Testing Your Advanced Script

Now that our script is complete, it's time to test it! You would typically run it in the background (e.g., ./monitor.sh &), then create test files in the $MONITOR_DIR and observe the $LOG_FILE and $ARCHIVE_DIR.

Here's a small script you can run to easily create a test file for your monitor.

#!/bin/bash
# create_test_file.sh

MONITOR_DIR="/tmp/coddykit_monitor"
mkdir -p "$MONITOR_DIR" # Ensure dir exists for this test script

# Create a unique test file
TEST_FILE="$MONITOR_DIR/test_log_$(date +%Y%m%d_%H%M%S).txt"
echo "This is a test entry." > "$TEST_FILE"
echo "Another line for testing." >> "$TEST_FILE"
echo "Created new test file: $TEST_FILE"

Script Logic Review

Our advanced script combines many concepts. Let's quickly review some of the core commands and structures we used.

Think about the role each command plays in achieving our project goal.

Beyond: Scheduling & Customization

This script is designed to run continuously. In a real-world scenario, you might run it in the background using nohup ./script.sh & or, more commonly, schedule it to run periodically using Cron.

Remember, Cron is perfect for automating tasks like this. You could schedule it to run every few minutes, checking the directory and processing files as needed.

Consider customizing it to:

Process different file types.
Perform different actions (e.g., compress files, upload to cloud).
Add more advanced error reporting (e.g., email alerts).

Project Recap & Next Steps

Congratulations! You've just completed an advanced Bash scripting project, integrating a wide array of Linux commands and scripting techniques.

You learned to:

Structure a robust, long-running script.
Use variables for configuration.
Implement loops for continuous monitoring.
Combine commands like find, wc, and mv.
Add error handling and graceful exits with trap.

This project is a solid foundation for building more complex automation solutions. Keep experimenting and building!

Frequently asked questions

Is the “Advanced Scripting Project” lesson free?

Yes — the full text of “Advanced Scripting Project” is free to read here on the web, and the DevOps Bootcamp course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the DevOps Bootcamp course, upgrade to CoddyKit PRO.

What will I learn in “Advanced Scripting Project”?

Work through a comprehensive project that integrates all learned concepts, from command-line tools to advanced Bash scripting techniques. You practise DevOps Bootcamp with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start DevOps Bootcamp?

No prior experience is required. DevOps Bootcamp on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Advanced Scripting Project” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this DevOps Bootcamp lesson?

Yes. Every DevOps Bootcamp lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to DevOps Bootcamp