CloudWatch Logs and Log Insights

Amazon CloudWatch Logs is a fully managed log aggregation service that collects, stores, and analyses log data from AWS services and your own applications. Unlike traditional log management where you SSH into servers to read files, CloudWatch Logs centralises all logs in a durable, encrypted store with built-in querying, filtering, metric extraction, and retention management. It integrates natively with EC2, Lambda, ECS, EKS, API Gateway, CloudTrail, and VPC Flow Logs.

Logs are organised hierarchically: a log group is a named container for logs that share the same retention and access control settings (for example, /aws/lambda/my-function). Within a log group, each log stream represents a single source of sequential log events — one EC2 instance, one Lambda container, or one ECS task. A log group can have thousands of streams, but each stream belongs to exactly one log group.

AWS Lambda automatically sends all stdout and stderr output to CloudWatch Logs — you just need the Lambda execution role to include logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents. For EC2 instances, you install the CloudWatch Agent and configure which log files to tail (for example, /var/log/nginx/access.log), what log group to write to, and what pattern identifies individual log entries.

CloudWatch Logs Insights is an interactive query engine that lets you search and analyse log data using a purpose-built query language. You can run queries across one or more log groups, filter by time range, extract fields from structured or unstructured log lines, aggregate data, and visualise results as bar charts or time series. Queries run on demand and are charged per GB of data scanned — structuring logs as JSON reduces scan volume significantly.

The Logs Insights query language uses pipe-separated commands: fields selects which fields to display, filter narrows events (supports regex with like and =~), stats aggregates data, sort orders results, and limit caps the number of results. The service automatically discovers fields in JSON-formatted log lines. For unstructured logs, you can parse values using parse with glob or regex patterns.

Metric filters continuously scan incoming log events for patterns and increment a custom CloudWatch metric each time a match is found. For example, you can create a metric filter on a Lambda log group that increments an ErrorCount metric every time a log line contains ERROR. You then create a CloudWatch alarm on this metric to alert your team. This converts your logs into actionable metrics without storing results in a separate database.

A subscription filter streams log events in near real time to another service for processing or storage. You can stream to AWS Lambda (to transform and route logs), Amazon Kinesis Data Streams (for high-volume log processing), or Amazon Kinesis Firehose (to deliver to S3, OpenSearch, or Splunk). Each log group supports up to two subscription filters. Cross-account log streaming requires a destination with an appropriate resource policy.

By default, CloudWatch Logs stores data indefinitely, which can become expensive. Set retention policies on every log group to automatically delete old log events. Common retention periods are 7 days for debugging logs, 30 days for application logs, and 1-5 years for audit logs. You can export log groups to S3 for long-term, low-cost archiving using the create-export-task API, though exports are batch operations (not real time).

VPC Flow Logs capture metadata about IP traffic flowing to and from network interfaces in your VPC, subnets, or individual ENIs. They are stored in CloudWatch Logs or S3. Flow logs include source IP, destination IP, port, protocol, packet count, byte count, action (ACCEPT or REJECT), and timestamp. They are invaluable for troubleshooting security group and NACL rules, detecting port scans, and auditing traffic patterns — but they do NOT capture actual packet payloads.

CloudWatch Logs Insights is particularly powerful for VPC Flow Log analysis. You can quickly identify rejected traffic, find the top talkers by byte volume, detect unusual connections, or isolate traffic to a specific instance. The flow log format is space-separated by default, but publishing in JSON format to CloudWatch Logs makes field extraction automatic in Insights queries.

CloudWatch Logs encrypts all data at rest by default using AWS-managed keys. For enhanced security — particularly in regulated environments — you can associate a customer-managed KMS key with a log group. This gives you control over key rotation and access, and enables you to revoke access to all log data by disabling the key. The KMS key must have a key policy that grants CloudWatch Logs permission to use it for encryption and decryption.

Test your understanding of AWS Solutions Architect (SAA-C03) concepts from this lesson.

In this lesson you learned: log groups and streams organise logs by source with configurable retention, Logs Insights lets you query and aggregate log data with a powerful pipe-based query language, and metric filters convert log patterns into CloudWatch metrics that can trigger alarms. Next up we explore CloudWatch Dashboards and Container Insights for operational visibility.