Kinesis Streams, Firehose, and Real-Time Analytics
Ingest streaming data with Kinesis Data Streams, deliver it to S3 or Redshift with Firehose, and analyse it in real time with Managed Service for Apache Flink.
Kinesis Streams, Firehose, and Real-Time Analytics is a free AWS Solutions Architect lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AWS Solutions Architect learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
The Kinesis Family Overview
Amazon Kinesis is a family of services for collecting, processing, and analysing real-time streaming data. The three core services are: Kinesis Data Streams (low-latency, custom processing), Kinesis Data Firehose (fully managed delivery to S3/Redshift/OpenSearch), and Managed Service for Apache Flink (formerly Kinesis Data Analytics) for real-time SQL and Flink processing. Each service targets a different point in the streaming pipeline.
Kinesis Data Streams Architecture
A Kinesis Data Stream is a durable, ordered log partitioned into shards. Each shard provides 1 MB/s write throughput and 2 MB/s read throughput. Data records are retained for 24 hours by default (extendable to 7 days or 365 days). Producers write records to a stream; consumers — Lambda, KCL applications, Firehose, or Flink — read from one or more shards in parallel. Records are immutable once written.
# Create a Kinesis Data Stream with 4 shards
aws kinesis create-stream \
--stream-name clickstream \
--shard-count 4
# Put a record into the stream
aws kinesis put-record \
--stream-name clickstream \
--partition-key 'user-123' \
--data 'eyJldmVudCI6ICJjbGljayJ9'Shards, Throughput, and Scaling
The number of shards determines a stream's total throughput. You can split a shard to double throughput or merge two shards to reduce cost. Use Enhanced Fan-Out to give each registered consumer its own 2 MB/s read throughput independently of other consumers, eliminating read throttling when multiple applications consume the same stream. Monitor GetRecords.IteratorAgeMilliseconds to detect consumer lag.
# Split shard to increase throughput
aws kinesis split-shard \
--stream-name clickstream \
--shard-to-split shardId-000000000001 \
--new-starting-hash-key 170141183460469231731687303715884105728
# Register an enhanced fan-out consumer
aws kinesis register-stream-consumer \
--stream-arn arn:aws:kinesis:us-east-1:123456789012:stream/clickstream \
--consumer-name analytics-appKinesis Data Firehose: Managed Delivery
Kinesis Data Firehose is a fully managed service that captures, transforms, and delivers streaming data to destinations including S3, Amazon Redshift, Amazon OpenSearch Service, Splunk, and HTTP endpoints. There are no shards to manage — Firehose scales automatically. You configure a buffer size (1–128 MB) and buffer interval (60–900 seconds); Firehose delivers whenever either limit is reached first.
# Create a Firehose delivery stream to S3
aws firehose create-delivery-stream \
--delivery-stream-name clickstream-to-s3 \
--s3-destination-configuration '{
"RoleARN": "arn:aws:iam::123456789012:role/FirehoseRole",
"BucketARN": "arn:aws:s3:::my-data-lake-123",
"Prefix": "landing/clickstream/year=!{timestamp:yyyy}/month=!{timestamp:MM}/",
"BufferingHints": {"SizeInMBs": 64, "IntervalInSeconds": 300},
"CompressionFormat": "GZIP"
}'Firehose Data Transformation with Lambda
Firehose can invoke a Lambda function on each batch of records before delivery to transform, enrich, or filter the data in-flight. Common use cases include converting JSON to Parquet (via Glue schema), masking PII fields, or dropping low-value events. Records that fail transformation are optionally sent to a separate S3 error prefix for reprocessing, so no data is lost.
# Lambda transform function signature for Firehose
def lambda_handler(event, context):
output = []
for record in event['records']:
import base64, json
payload = json.loads(base64.b64decode(record['data']))
# Drop events with no user_id
if not payload.get('user_id'):
output.append({'recordId': record['recordId'], 'result': 'Dropped', 'data': record['data']})
else:
output.append({'recordId': record['recordId'], 'result': 'Ok', 'data': record['data']})
return {'records': output}Managed Service for Apache Flink
Amazon Managed Service for Apache Flink (formerly Kinesis Data Analytics) runs Apache Flink applications on fully managed infrastructure. Use it for stateful, real-time analytics: sliding window aggregations, anomaly detection, pattern matching over event sequences, and joining streaming data with reference tables. You write Flink code in Java, Python, or Scala and Flink manages checkpoints and exactly-once state.
# Flink SQL-style tumbling window (conceptual)
# Count page views per URL every 5 minutes
CREATE TABLE clickstream (
url STRING,
event_time TIMESTAMP(3),
WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
) WITH ('connector' = 'kinesis', 'stream' = 'clickstream', ...);
SELECT
url,
COUNT(*) AS views,
TUMBLE_START(event_time, INTERVAL '5' MINUTE) AS window_start
FROM clickstream
GROUP BY url, TUMBLE(event_time, INTERVAL '5' MINUTE);Choosing Between Streams and Firehose
For the SAA-C03 exam, know the decision criteria. Use Kinesis Data Streams when you need sub-second latency, multiple consumers reading simultaneously, or custom processing logic with full control over retention and replay. Use Firehose when you just need to reliably deliver streaming data to S3, Redshift, or OpenSearch with minimal code, optional in-flight transformation, and automatic scaling at higher latency (60+ seconds).
Kinesis vs SQS: The Classic Exam Choice
A common SAA-C03 question asks you to choose between Kinesis and SQS. Key differences: Kinesis preserves message order within a shard, supports multiple consumers reading the same data simultaneously, and retains records for replay. SQS removes messages after they are consumed (no replay), FIFO mode guarantees strict ordering, and it is better for decoupling microservices. If the scenario mentions real-time analytics or replay, choose Kinesis.
Kinesis Producers: SDK and KPL
The Kinesis Producer Library (KPL) is a high-throughput client for writing to Kinesis Data Streams from applications. KPL automatically aggregates multiple small records into a single API call (up to 1 MB) and handles retries with back-off. This dramatically reduces the per-record PUT cost and increases throughput per shard. Use KPL for high-volume producers like web clickstreams, IoT sensors, or log pipelines.
# Basic KPL usage (Java pseudocode, no backticks)
KinesisProducer producer = new KinesisProducer();
byte[] data = 'hello world'.getBytes();
ByteBuffer buf = ByteBuffer.wrap(data);
// addUserRecord handles aggregation and retry internally
ListenableFuture future = producer.addUserRecord('clickstream', 'partitionKey', buf);
Futures.addCallback(future, new FutureCallback() { ... });Firehose to Redshift Pattern
A common architecture is to use Firehose as a managed pipeline from Kinesis streams or direct producers into Amazon Redshift for data warehousing. Firehose first writes data to an intermediate S3 staging bucket, then issues a COPY command to load the data into Redshift. This is the most efficient way to bulk-load streaming data into Redshift — direct row-by-row inserts into Redshift would be extremely slow due to per-row overhead.
# Firehose Redshift destination (CLI snippet)
--redshift-destination-configuration '{
"RoleARN": "arn:aws:iam::123456789012:role/FirehoseRole",
"ClusterJDBCURL": "jdbc:redshift://cluster.xyz.us-east-1.redshift.amazonaws.com:5439/sales",
"CopyCommand": {
"DataTableName": "clickevents",
"CopyOptions": "JSON 'auto'"
},
"Username": "firehose_user",
"Password": "{{resolve:secretsmanager:redshift-pw}}",
"S3Configuration": {
"RoleARN": "...",
"BucketARN": "arn:aws:s3:::firehose-staging"
}
}'Monitoring Kinesis with CloudWatch
Key CloudWatch metrics for Kinesis: IncomingBytes and IncomingRecords to measure producer throughput, GetRecords.IteratorAgeMilliseconds to measure consumer lag (a high value means consumers cannot keep up), and WriteProvisionedThroughputExceeded to detect when producers are hitting shard limits. Set CloudWatch alarms on iterator age and provisioned throughput exceeded to trigger Auto Scaling of shards via Application Auto Scaling.
# CloudWatch alarm on high consumer lag
aws cloudwatch put-metric-alarm \
--alarm-name kinesis-high-lag \
--metric-name GetRecords.IteratorAgeMilliseconds \
--namespace AWS/Kinesis \
--dimensions Name=StreamName,Value=clickstream \
--statistic Maximum \
--period 60 \
--threshold 60000 \
--comparison-operator GreaterThanThreshold \
--evaluation-periods 3 \
--alarm-actions arn:aws:sns:us-east-1:123456789012:ops-alertsQuick Check
Test your understanding of AWS Solutions Architect (SAA-C03) concepts from this lesson.
Lesson Recap
In this lesson you learned: Kinesis Data Streams provides durable, ordered, sharded streaming with multiple consumers and replay capability, Kinesis Data Firehose offers fully managed, no-code delivery to S3, Redshift, and OpenSearch, and Managed Service for Apache Flink enables stateful real-time analytics on streams. Next up we explore the 7 Rs of cloud migration strategy.
Frequently asked questions
Is the “Kinesis Streams, Firehose, and Real-Time Analytics” lesson free?
Yes — the full text of “Kinesis Streams, Firehose, and Real-Time Analytics” is free to read here on the web, and the AWS Solutions Architect course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AWS Solutions Architect course, upgrade to CoddyKit PRO.
What will I learn in “Kinesis Streams, Firehose, and Real-Time Analytics”?
Ingest streaming data with Kinesis Data Streams, deliver it to S3 or Redshift with Firehose, and analyse it in real time with Managed Service for Apache Flink. You practise AWS Solutions Architect with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AWS Solutions Architect?
No prior experience is required. AWS Solutions Architect on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Kinesis Streams, Firehose, and Real-Time Analytics” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AWS Solutions Architect lesson?
Yes. Every AWS Solutions Architect lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Building a Data Lake on S3
- AWS Glue: ETL and Data Catalogue
- Amazon Athena: Serverless SQL on S3
- Kinesis Streams, Firehose, and Real-Time Analytics