Collecting Logs into CloudWatch

Amazon CloudWatch Logs is the managed service that collects, stores, and searches log data from your AWS resources and applications. Instead of logs sitting on individual servers where they can be lost when an instance terminates, they stream to a central, durable place. For a security engineer, this centralization is the whole point: you cannot detect or investigate what you cannot see.

A log group is a container that holds related log data and defines shared settings. Typically one log group represents one application or service, such as /aws/lambda/my-function. Two settings matter most for security: the retention period, which controls how long logs are kept before automatic deletion, and the optional KMS key used to encrypt the data at rest.

Inside a log group, a log stream is a sequence of events from a single source, like one EC2 instance or one Lambda invocation environment. Events arrive with a timestamp and a message. Splitting by stream keeps sources separate so you can trace activity back to a specific origin while still grouping everything logically under one log group.

EC2 instances and on-premises servers do not send logs automatically. The unified CloudWatch agent is installed on the host to ship operating-system, application, and custom log files into CloudWatch Logs. It also collects extra metrics such as memory and disk usage that the hypervisor cannot see. The agent needs an IAM role granting it permission to write logs.

Many AWS services log directly to CloudWatch with no agent at all. Lambda writes function output, VPC Flow Logs can target a log group, and Route 53 can log DNS queries. CloudTrail can also deliver events to CloudWatch Logs in addition to S3. Knowing which sources are agentless versus agent-based is a common exam distinction.

By default log groups keep data forever, which quietly grows your bill. You set retention per log group, from one day up to ten years. Security and compliance often demand long retention for audit trails, while noisy debug logs can expire quickly. Balancing the two keeps both auditors and your finance team satisfied.

CloudWatch Logs encrypts data at rest by default, but you can associate a customer managed KMS key with a log group for tighter control. This lets you govern exactly who can read the logs through the key policy and produces an auditable record of key usage. Sensitive logs containing credentials or PII benefit most from this.

A subscription filter streams matching log events in near real time to another destination such as Lambda, Kinesis Data Streams, or Amazon OpenSearch. Security teams use this to forward logs to a SIEM or to a central logging account the moment events arrive, rather than batch-exporting later. It turns CloudWatch Logs into a live pipeline.

For long-term, low-cost archival you can export log data to Amazon S3. S3 offers cheaper storage, lifecycle rules to Glacier, and Object Lock for immutability. A common pattern is short retention in CloudWatch for fast search plus an export to a locked-down S3 bucket for the years-long compliance record.

CloudWatch Logs Insights is a purpose-built query language for searching log groups interactively. You can filter, aggregate, and sort across huge volumes without exporting anything. During an investigation it lets you quickly answer questions like which IP triggered the most errors, making it a fast first step before moving to Athena for deeper analysis.

The logging pipeline is itself a target. Grant the CloudWatch agent role only logs:CreateLogStream and logs:PutLogEvents, never broad permissions. Protect log groups so attackers cannot delete them to hide activity, and forward critical logs to a separate account quickly. A log you control end to end is evidence you can trust.

Test your grasp of CloudWatch Logs collection.

CloudWatch Logs centralizes log data into log groups made of log streams. EC2 and on-prem hosts need the unified agent, while many services integrate natively. Set retention to control cost, attach a KMS key for sensitive data, and use subscription filters or S3 export to feed a SIEM or archive. Locking down the pipeline keeps your evidence trustworthy.