Kinesis Data Streams for Real-Time Event Processing
Produce and consume high-throughput event streams with Kinesis Data Streams, manage shards for throughput, and use Lambda as a consumer.
Kinesis Data Streams Core Concepts
Kinesis Data Streams (KDS) is a durable, ordered, real-time data streaming service. Data is organised into a stream composed of one or more shards. Each shard is an ordered sequence of data records. Producers put records onto shards using a partition key that determines which shard receives the record. Consumers read records from shards, processing them in the order they arrived within each shard.
Shard Capacity and Throughput Limits
Each shard supports 1 MB/s or 1,000 records/s write throughput and 2 MB/s read throughput (shared among all standard consumers on that shard). Total stream capacity scales linearly with shard count. Use the formula: shards_needed = max(write_MB_per_s / 1, read_MB_per_s / 2). If producers hit the write limit, ProvisionedThroughputExceededException errors appear — resolve by splitting shards or distributing partition keys more evenly.
# Calculate shards needed for a stream:
# - Ingest rate: 5 MB/s writes
# - Read rate: 3 consumers x 2 MB/s = 6 MB/s reads
# shards = max(5/1, 6/2) = max(5, 3) = 5 shards needed
aws kinesis create-stream \
--stream-name iot-telemetry \
--shard-count 5All lessons in this course
- EventBridge: Event Bus and Rules
- Step Functions: Orchestrating Serverless Workflows
- Kinesis Data Streams for Real-Time Event Processing
- Choreography vs Orchestration Patterns