AWS for Backend Developers (EC2, S3, RDS, Lambda) · Lesson

CloudWatch Logs and Log Groups

Aggregate, monitor, and store logs from various AWS services and custom applications for troubleshooting and auditing.

CloudWatch Logs and Log Groups is a free AWS for Backend Developers (EC2, S3, RDS, Lambda) lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AWS for Backend Developers (EC2, S3, RDS, Lambda) learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

CloudWatch Logs Intro

Imagine trying to fix a problem in your application without knowing what happened! That's where logs come in.

CloudWatch Logs is an AWS service that helps you collect, monitor, and store logs from your applications and various AWS services.

It's crucial for troubleshooting, auditing, and understanding how your systems are performing.

Log Groups Explained

At the core of CloudWatch Logs is the concept of a Log Group.

Think of a Log Group as a dedicated folder or container for logs that share the same characteristics, like logs from a specific application or service.

For example, all logs from your 'WebApp-Production' service might go into one Log Group.

Log Streams within Groups

Inside each Log Group, you'll find Log Streams.

A Log Stream is a sequence of log events from a single source within that Log Group.

For an EC2 instance, each instance might have its own Log Stream.
For a Lambda function, each invocation or version could generate a Log Stream.

They help you pinpoint the exact source of your log data.

Setting Up Log Groups

Many AWS services automatically create Log Groups for you when they generate logs, like AWS Lambda or AWS Fargate.

For custom applications, you'll typically create a Log Group manually and then configure your application or the CloudWatch Agent to send logs to it.

You can create and manage Log Groups via the AWS Management Console, AWS CLI, or SDKs.

Ingesting EC2 Logs

To send logs from applications running on an EC2 instance to CloudWatch Logs, you use the CloudWatch Agent.

Install the agent on your EC2 instance.
Configure it to monitor specific log files (e.g., /var/log/myapp.log).
The agent then streams these logs to a specified Log Group and Log Stream in CloudWatch Logs.

Lambda's Automatic Logs

AWS Lambda automatically integrates with CloudWatch Logs! Any output your Lambda function generates (e.g., using print() in Python or console.log() in Node.js) is sent to CloudWatch Logs.

Try running this simple Python Lambda function and check its logs:

import json

def lambda_handler(event, context):
    print("INFO: Lambda function execution started.")
    
    # Simulate processing an event
    if 'detail' in event and 'message' in event['detail']:
        print(f"DEBUG: Processing message: {event['detail']['message']}")
    else:
        print("DEBUG: No specific message in event.")
        
    # Simulate an error condition based on input
    if 'statusCode' in event and event['statusCode'] >= 400:
        print("ERROR: An error status was detected in the input event!")
        return {
            'statusCode': event['statusCode'],
            'body': json.dumps('Error processed!')
        }
    
    print("INFO: Lambda function execution finished successfully.")
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

Finding Info in Logs

Once your logs are in CloudWatch Logs, you can easily search and filter them.

Search: Look for specific keywords, phrases, or patterns across all your log events.
Filter: Narrow down results by time range, Log Stream, or even specific fields if your logs are structured (e.g., JSON).

This helps you quickly diagnose issues or find relevant events.

Managing Log Retention

Logs can consume a lot of storage, and you might have compliance requirements for how long to keep them.

Log Retention Policies allow you to define how long CloudWatch Logs should store your log events.

Options range from 'Never Expire' to 1 day, 30 days, 1 year, etc.
Setting appropriate retention helps manage costs and adhere to data policies.

Log Metrics Filters

Beyond just viewing logs, you can extract meaningful data from them using Metric Filters.

A Metric Filter defines a pattern to look for in your log events (e.g., the word "ERROR" or a specific status code).

When the pattern is matched, CloudWatch increments a custom metric, which you can then use to create alarms (as seen in the previous lesson!).

Log Group Basics

Time for a quick check on CloudWatch Logs!

Which statement best describes the relationship between CloudWatch Log Groups and Log Streams?

Logs Recap

Great job! You've learned the essentials of CloudWatch Logs.

We covered:

Log Groups for organizing logs.
Log Streams for individual log sources.
How to ingest logs from services like EC2 and Lambda.
How to search and filter logs efficiently.
Configuring retention policies and creating metric filters from logs.

Next, we'll explore CloudTrail for event-driven monitoring and auditing.

Frequently asked questions

Is the “CloudWatch Logs and Log Groups” lesson free?

Yes — the full text of “CloudWatch Logs and Log Groups” is free to read here on the web, and the AWS for Backend Developers (EC2, S3, RDS, Lambda) course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AWS for Backend Developers (EC2, S3, RDS, Lambda) course, upgrade to CoddyKit PRO.

What will I learn in “CloudWatch Logs and Log Groups”?

Aggregate, monitor, and store logs from various AWS services and custom applications for troubleshooting and auditing. You practise AWS for Backend Developers (EC2, S3, RDS, Lambda) with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AWS for Backend Developers (EC2, S3, RDS, Lambda)?

No prior experience is required. AWS for Backend Developers (EC2, S3, RDS, Lambda) on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “CloudWatch Logs and Log Groups” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AWS for Backend Developers (EC2, S3, RDS, Lambda) lesson?

Yes. Every AWS for Backend Developers (EC2, S3, RDS, Lambda) lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AWS for Backend Developers (EC2, S3, RDS, Lambda)