0Pricing
Elixir & Phoenix: Scalable Backend Development · Lesson

GenStage and Backpressure Pipelines

Build demand-driven data pipelines with GenStage, where consumers control flow through backpressure to prevent overload in distributed systems.

GenStage and Backpressure Pipelines is a free Elixir & Phoenix: Scalable Backend Development lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Elixir & Phoenix: Scalable Backend Development learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

The Overload Problem

When a fast producer feeds a slow consumer, messages pile up and memory explodes. Backpressure solves this: consumers signal how much work they can handle, and producers send only that much.

What is GenStage?

GenStage is an Elixir behaviour for building staged, demand-driven pipelines. Data flows from producers to consumers, with optional producer-consumer stages in between.

The Three Stage Types

Every GenStage process plays one role:

  • Producer: emits events on demand
  • ProducerConsumer: receives, transforms, and re-emits
  • Consumer: receives and processes events

Defining a Producer

A producer implements init/1 and handle_demand/2. It returns events only when consumers ask for them.

defmodule Counter do
  use GenStage
  def init(start), do: {:producer, start}
  def handle_demand(demand, state) do
    events = Enum.to_list(state..(state + demand - 1))
    {:noreply, events, state + demand}
  end
end

Defining a Consumer

A consumer implements handle_events/3. It returns no events itself, only acknowledging the demand it processed.

defmodule Printer do
  use GenStage
  def init(_), do: {:consumer, :ok}
  def handle_events(events, _from, state) do
    for e <- events, do: IO.inspect(e)
    {:noreply, [], state}
  end
end

Subscribing Stages

Connect a consumer to a producer with GenStage.sync_subscribe/2. The max_demand option caps how many events flow at once.

{:ok, prod} = GenStage.start_link(Counter, 0)
{:ok, cons} = GenStage.start_link(Printer, :ok)
GenStage.sync_subscribe(cons, to: prod, max_demand: 10)

How Backpressure Works

The consumer requests up to max_demand events. The producer can never send more than was requested, so a slow consumer naturally throttles a fast producer. No buffering blowup.

Producer-Consumer Stages

A middle stage transforms data. It implements handle_events/3 but returns transformed events for the next stage.

def handle_events(events, _from, state) do
  doubled = Enum.map(events, &(&1 * 2))
  {:noreply, doubled, state}
end

ConsumerSupervisor

For concurrent processing, ConsumerSupervisor spawns a short-lived child process per event, bounded by demand. This parallelizes work while keeping backpressure intact.

Flow for Parallel Pipelines

The Flow library builds on GenStage to offer map/reduce-style parallel data processing with partitioning — ideal for crunching large collections across cores.

File.stream!("big.csv")
|> Flow.from_enumerable()
|> Flow.map(&parse_line/1)
|> Flow.partition()
|> Enum.to_list()

When to Reach for GenStage

Use GenStage when you have:

  • A rate mismatch between data source and processing
  • Streaming data that must not overwhelm memory
  • Multi-stage transformation pipelines

Quick Check

Test your GenStage knowledge.

Recap

You learned demand-driven pipelines:

  • GenStage has producers, producer-consumers, and consumers
  • Consumers request bounded demand for backpressure
  • Subscribe stages with sync_subscribe and max_demand
  • ConsumerSupervisor parallelizes per-event work
  • Flow builds parallel map/reduce pipelines on top

Backpressure keeps distributed systems stable under load.

Frequently asked questions

Is the “GenStage and Backpressure Pipelines” lesson free?

Yes — the full text of “GenStage and Backpressure Pipelines” is free to read here on the web, and the Elixir & Phoenix: Scalable Backend Development course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Elixir & Phoenix: Scalable Backend Development course, upgrade to CoddyKit PRO.

What will I learn in “GenStage and Backpressure Pipelines”?

Build demand-driven data pipelines with GenStage, where consumers control flow through backpressure to prevent overload in distributed systems. You practise Elixir & Phoenix: Scalable Backend Development with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Elixir & Phoenix: Scalable Backend Development?

No prior experience is required. Elixir & Phoenix: Scalable Backend Development on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “GenStage and Backpressure Pipelines” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Elixir & Phoenix: Scalable Backend Development lesson?

Yes. Every Elixir & Phoenix: Scalable Backend Development lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Distributed Elixir and Clustering
  2. Advanced Supervisor Strategies
  3. Dynamic Supervisors and Registry
  4. GenStage and Backpressure Pipelines
← Back to Elixir & Phoenix: Scalable Backend Development