0Pricing
Apache Kafka & Stream Processing Fundamentals · Lesson

Consumer Lag Tracking & Alerting

Learn what consumer lag means, how to measure it with built-in tools, and how to alert on it before consumers fall dangerously behind.

Consumer Lag Tracking & Alerting is a free Apache Kafka & Stream Processing Fundamentals lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Apache Kafka & Stream Processing Fundamentals learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

What Is Consumer Lag?

Consumer lag is the difference between the latest offset produced to a partition (the log-end offset) and the offset a consumer group has committed.

Lag tells you how far behind your consumers are. Rising lag means consumers can't keep up with producers.

Why Lag Matters

High lag has real consequences:

  • Stale data downstream (dashboards, alerts, ML features).
  • Risk of hitting retention and losing unconsumed messages.
  • A signal of undersized consumers or a stuck partition.

Checking Lag from the CLI

The fastest way to see lag is kafka-consumer-groups.sh with --describe.

kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --group order-processors

Reading the Output

The describe output shows per-partition columns:

  • CURRENT-OFFSET — last committed offset.
  • LOG-END-OFFSET — newest offset in the partition.
  • LAG — the difference between them.

Sum LAG across partitions for total group lag.

Lag via JMX Metrics

Each consumer exposes lag through JMX under the consumer-fetch-manager-metrics group.

  • records-lag-max — max lag across assigned partitions.
  • records-lag — per-partition lag.

These are client-side and update in near real time.

Burrow & Kafka Exporter

For production monitoring, dedicated tools poll lag for all groups:

  • Burrow — evaluates lag trend and reports group status (OK/WARN/ERR).
  • kafka_exporter — exposes lag as Prometheus metrics.

Prometheus Lag Metric

With kafka_exporter, lag is available as a labeled time series you can graph and alert on in Grafana.

# Example Prometheus metric
kafka_consumergroup_lag{consumergroup="order-processors",topic="orders",partition="0"} 1423

Writing a Lag Alert

Alert on sustained high lag, not momentary spikes. A common rule fires when total lag exceeds a threshold for several minutes.

# Prometheus alert rule
- alert: HighConsumerLag
  expr: sum(kafka_consumergroup_lag{consumergroup="order-processors"}) > 10000
  for: 5m
  labels:
    severity: warning

Lag Rate vs. Absolute Lag

Absolute lag alone can mislead — 10,000 messages may be seconds of data for a fast topic.

Track the rate of change: if lag keeps growing, consumers are losing the race even if current numbers look fine.

Reducing Lag

When lag climbs, your options include:

  • Add consumers (up to the partition count).
  • Increase partitions to allow more parallelism.
  • Tune max.poll.records and processing efficiency.
  • Check for a slow or stuck partition / poison message.

Operational Best Practices

Make lag a first-class signal:

  • Dashboard total and per-partition lag for every critical group.
  • Alert on lag trend, not just thresholds.
  • Keep lag well below the retention window so you never lose data.

Quick Check

Test your understanding of consumer lag.

Recap

You learned to track and alert on consumer lag.

  • Lag = log-end offset minus committed offset.
  • Inspect it via kafka-consumer-groups.sh, JMX, Burrow, or kafka_exporter.
  • Alert on sustained lag and lag growth rate.
  • Reduce lag by scaling consumers/partitions and tuning processing.

Frequently asked questions

Is the “Consumer Lag Tracking & Alerting” lesson free?

Yes — the full text of “Consumer Lag Tracking & Alerting” is free to read here on the web, and the Apache Kafka & Stream Processing Fundamentals course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Apache Kafka & Stream Processing Fundamentals course, upgrade to CoddyKit PRO.

What will I learn in “Consumer Lag Tracking & Alerting”?

Learn what consumer lag means, how to measure it with built-in tools, and how to alert on it before consumers fall dangerously behind. You practise Apache Kafka & Stream Processing Fundamentals with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Apache Kafka & Stream Processing Fundamentals?

No prior experience is required. Apache Kafka & Stream Processing Fundamentals on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Consumer Lag Tracking & Alerting” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Apache Kafka & Stream Processing Fundamentals lesson?

Yes. Every Apache Kafka & Stream Processing Fundamentals lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Command-Line Tools for Kafka
  2. Monitoring Kafka with JMX & Tools
  3. Security: Authentication & Authorization
  4. Consumer Lag Tracking & Alerting
← Back to Apache Kafka & Stream Processing Fundamentals