Kinesis Data Streams for Real-Time Event Processing

Kinesis Data Streams (KDS) is a durable, ordered, real-time data streaming service. Data is organised into a stream composed of one or more shards. Each shard is an ordered sequence of data records. Producers put records onto shards using a partition key that determines which shard receives the record. Consumers read records from shards, processing them in the order they arrived within each shard.

Each shard supports 1 MB/s or 1,000 records/s write throughput and 2 MB/s read throughput (shared among all standard consumers on that shard). Total stream capacity scales linearly with shard count. Use the formula: shards_needed = max(write_MB_per_s / 1, read_MB_per_s / 2). If producers hit the write limit, ProvisionedThroughputExceededException errors appear — resolve by splitting shards or distributing partition keys more evenly.

The partition key is a string that Kinesis hashes (MD5) to determine which shard receives a record. A well-chosen partition key distributes records evenly across shards (hot shard prevention). For IoT: use device ID. For clickstreams: use session ID or user ID. Avoid low-cardinality keys (e.g., country name with only 5 values) as they cause hot shards where one shard receives disproportionate write traffic while others sit idle.

Standard consumers share the 2 MB/s read throughput per shard using GetRecords with polling. If you have 3 consumers on a shard each needs 2 MB/s, they will be throttled sharing 2 MB/s total. Enhanced Fan-Out (EFO) gives each registered consumer its own dedicated 2 MB/s read pipe via a persistent HTTP/2 push connection (SubscribeToShard). EFO adds a per-consumer-shard-hour cost but eliminates read contention entirely.

Lambda integrates natively with Kinesis Data Streams via an Event Source Mapping. Lambda polls the stream, reads batches of records, and invokes your function. Configure BatchSize (1–10,000 records), StartingPosition (TRIM_HORIZON for oldest, LATEST for newest), and BisectBatchOnFunctionError to split failed batches. Parallelisation Factor (1–10) lets Lambda launch multiple concurrent invocations per shard to keep up with fast-moving streams.

The Kinesis Client Library (KCL) is an application framework for building robust Java (or multi-language via MultiLangDaemon) Kinesis consumers. KCL handles shard enumeration, lease management (distributing shards among worker instances), checkpointing progress to DynamoDB, and graceful shard splits/merges. Each KCL worker processes one or more shards, and KCL automatically rebalances shards as workers scale out or fail. KCL is preferred over raw SDK polling for production consumer applications.

Kinesis Data Streams stores records for 24 hours by default (extendable to 7 days or up to 365 days with Long-Term Retention at additional cost). Unlike SQS, consumed records are not deleted after consumption — they remain available until retention expires. This enables multiple consumers to read the same records independently, and allows replay by resetting a consumer's checkpoint to an earlier sequence number — invaluable for bug fixes or backfilling new services.

Kinesis Data Streams supports two capacity modes. Provisioned mode: you manage the number of shards manually and pay per shard-hour. On-Demand mode: Kinesis automatically scales shard capacity based on incoming throughput (up to 200 MB/s write and 400 MB/s read by default) and you pay per GB of data written and retrieved. On-demand mode is ideal for variable or unpredictable traffic patterns where you do not want to manage shard scaling.

Kinesis guarantees ordering within a shard — records with the same partition key always go to the same shard and are read in the order they were written. However, there is no ordering guarantee across shards. If your application requires global ordering across all records, use a single shard (limiting throughput to 1 MB/s) or redesign so ordering is only needed within a partition key group (e.g., per-device ordering). This is a common exam distinction versus SQS FIFO which provides strict deduplication and ordering.

Kinesis Data Streams encrypts records server-side at rest using AWS KMS (CMK or AWS-managed key) when server-side encryption is enabled. All data in transit is encrypted with TLS. For applications running in a VPC that should not send stream data over the public internet, use a VPC Interface Endpoint (PrivateLink) for Kinesis so traffic stays entirely within the AWS network backbone — important for regulated workloads.

For the SAA-C03 exam, compare Kinesis Data Streams with alternatives. Kinesis vs SQS: Kinesis preserves order within a shard and supports multiple consumers reading the same data; SQS deletes messages after consumption. Kinesis vs MSK (Kafka): MSK is managed Apache Kafka — use it when you need Kafka protocol compatibility, advanced topic configurations, or are migrating from on-premises Kafka. Use Kinesis for AWS-native streaming with tighter integration to Lambda, Firehose, and Flink. Choose Kinesis unless the question specifically mentions Kafka or Kafka compatibility requirements.

Test your understanding of AWS Solutions Architect (SAA-C03) concepts from this lesson.

In this lesson you learned: Kinesis Data Streams provides sharded, ordered, durable streams with 24-hour default retention and replay capability, Enhanced Fan-Out gives each consumer dedicated 2 MB/s per shard eliminating read contention, and On-Demand mode auto-scales shards for unpredictable traffic. Next up we explore the choreography versus orchestration patterns in event-driven architecture.