0Pricing
Lua Academy · Lesson

Producer-Consumer Pattern

Implement a producer-consumer pipeline using coroutines.

Producer-Consumer Pattern is a free Lua Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Lua Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

The Pattern Explained

The producer-consumer pattern with coroutines: the producer generates data and yields each item; the consumer receives items one at a time. Neither needs to know the other's implementation — they communicate through yield/resume. This decouples data generation from data processing.

local function makeProducer(items)
  return coroutine.create(function()
    for _, item in ipairs(items) do
      coroutine.yield(item)
    end
  end)
end

local function consume(producer)
  while true do
    local ok, item = coroutine.resume(producer)
    if not ok or item == nil then break end
    print("Processing:", item)
  end
end

consume(makeProducer({10, 20, 30, 40}))

Infinite Producer

A producer coroutine can yield an infinite sequence. The consumer controls when to stop. This is perfect for streaming data sources: sensor readings, log events, generated sequences.

local function primes()
  return coroutine.wrap(function()
    local function isPrime(n)
      if n<2 then return false end
      for i=2, math.floor(math.sqrt(n)) do
        if n%i==0 then return false end
      end
      return true
    end
    local n = 2
    while true do
      if isPrime(n) then coroutine.yield(n) end
      n = n + 1
    end
  end)
end

local gen = primes()
for i = 1, 10 do
  io.write(gen() .. " ")
end
print()  -- 2 3 5 7 11 13 17 19 23 29

Consumer as Coroutine

The consumer can also be a coroutine, creating a fully symmetric producer-consumer pair. A third "driver" function orchestrates both. This is useful for buffered pipelines where producer and consumer have different rates.

local function producer()
  return coroutine.create(function(consumer)
    for i = 1, 5 do
      coroutine.resume(consumer, i * 10)
      coroutine.yield()  -- wait
    end
  end)
end

local consumer = coroutine.create(function()
  while true do
    local val = coroutine.yield()
    print("Received:", val)
  end
end)

coroutine.resume(consumer)  -- start, waits for data
local pro = producer()
for i = 1, 5 do
  coroutine.resume(pro, consumer)
end

Backpressure

Backpressure is when the consumer signals the producer to slow down. With coroutines, this is natural: the producer yields after each item, waiting for the consumer to request the next one. If the consumer is slow, the producer simply waits.

local function slowConsumer(gen)
  for v in gen do
    -- Simulate slow processing
    print("Processing", v, "...")
    -- In real code: os.execute("sleep 0.1")
  end
end

local function fastProducer(n)
  return coroutine.wrap(function()
    for i = 1, n do
      print("Producing", i)
      coroutine.yield(i)
    end
  end)
end

-- Producer automatically waits for consumer:
slowConsumer(fastProducer(4))

Batched Processing

A batching producer collects N items before yielding a batch. The consumer receives and processes complete batches. This reduces coordination overhead for high-frequency producers.

local function batchProducer(items, batchSize)
  return coroutine.wrap(function()
    local batch = {}
    for _, item in ipairs(items) do
      batch[#batch+1] = item
      if #batch >= batchSize then
        coroutine.yield(batch)
        batch = {}
      end
    end
    if #batch > 0 then coroutine.yield(batch) end
  end)
end

local data = {1,2,3,4,5,6,7,8,9,10}
for batch in batchProducer(data, 3) do
  print("Batch:", table.concat(batch, ","))
end
-- Batch: 1,2,3
-- Batch: 4,5,6
-- Batch: 7,8,9
-- Batch: 10

File Line Producer

Read a file lazily: a producer coroutine reads lines one at a time and yields each. The consumer processes lines without loading the entire file into memory.

local function lineProducer(path)
  return coroutine.wrap(function()
    local f = io.open(path, "r")
    if not f then return end
    for line in f:lines() do
      coroutine.yield(line)
    end
    f:close()
  end)
end

-- Process file line by line without loading all at once
local wordCount = 0
for line in lineProducer("data.txt") do
  for _ in line:gmatch("%S+") do
    wordCount = wordCount + 1
  end
end
print("Words:", wordCount)

Transform Stage

Insert a transform stage between producer and consumer. The transform reads from one coroutine and yields processed values to another. Build data pipelines by chaining these stages.

local function transform(source, fn)
  return coroutine.wrap(function()
    for v in source do
      coroutine.yield(fn(v))
    end
  end)
end

local numbers = coroutine.wrap(function()
  for i = 1, 6 do coroutine.yield(i) end
end)

local doubled = transform(numbers, function(n) return n*2 end)
local filtered = coroutine.wrap(function()
  for v in doubled do
    if v > 4 then coroutine.yield(v) end
  end
end)

for v in filtered do io.write(v.." ") end
print()  -- 6 8 10 12

Error Propagation in Pipelines

Errors in coroutine pipelines propagate via the dead state or via pcall. Wrap pipeline stages in pcall to catch and handle errors gracefully without crashing the whole pipeline.

local function safeStage(source, fn)
  return coroutine.wrap(function()
    for v in source do
      local ok, result = pcall(fn, v)
      if ok then
        coroutine.yield(result)
      else
        print("Error processing", v, ":", result)
      end
    end
  end)
end

local data = coroutine.wrap(function()
  for _, v in ipairs({4, -1, 9, 0, 16}) do coroutine.yield(v) end
end)

local results = safeStage(data, function(n)
  assert(n > 0, "non-positive")
  return math.sqrt(n)
end)

for v in results do print(v) end

Merge Producers

Combine multiple producers into one: round-robin among all alive producers, yielding each item in turn. Stop when all producers are exhausted.

local function merge(...)
  local producers = {...}
  return coroutine.wrap(function()
    while #producers > 0 do
      local alive = {}
      for _, co in ipairs(producers) do
        local ok, v = coroutine.resume(co)
        if ok and v ~= nil then
          coroutine.yield(v)
          alive[#alive+1] = co
        end
      end
      producers = alive
    end
  end)
end

local function src(items)
  return coroutine.create(function()
    for _,v in ipairs(items) do coroutine.yield(v) end
  end)
end

for v in merge(src({1,3,5}),src({2,4,6})) do
  io.write(v.." ")
end
print()  -- 1 2 3 4 5 6

Real-World Use Case

Log processing pipeline: producer reads log lines, filter stage keeps only error lines, transform stage extracts key info, consumer aggregates by error type. Each stage is an independent, testable coroutine.

local function makeFilter(src, pred)
  return coroutine.wrap(function()
    for line in src do
      if pred(line) then coroutine.yield(line) end
    end
  end)
end

local logs = coroutine.wrap(function()
  local entries = {
    "INFO: started",
    "ERROR: timeout",
    "DEBUG: connecting",
    "ERROR: auth failed",
  }
  for _, e in ipairs(entries) do coroutine.yield(e) end
end)

local errors = makeFilter(logs, function(l) return l:match("^ERROR") end)
for line in errors do print(line) end
-- ERROR: timeout
-- ERROR: auth failed

Quick Check

How does the producer-consumer coroutine pattern handle backpressure?

Recap: Producer-Consumer

Summary:

  • Producer yields items; consumer drives by calling resume
  • Infinite producers: while true do yield() end
  • Pipelines: chain transform coroutines between producer and consumer
  • Backpressure is automatic: producer waits for consumer
  • Batch, filter, merge stages as coroutine wrappers
  • Use pcall in pipeline stages for error isolation

Frequently asked questions

Is the “Producer-Consumer Pattern” lesson free?

Yes — the full text of “Producer-Consumer Pattern” is free to read here on the web, and the Lua Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Lua Academy course, upgrade to CoddyKit PRO.

What will I learn in “Producer-Consumer Pattern”?

Implement a producer-consumer pipeline using coroutines. You practise Lua Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Lua Academy?

No prior experience is required. Lua Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Producer-Consumer Pattern” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Lua Academy lesson?

Yes. Every Lua Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Creating Coroutines
  2. resume and yield
  3. Coroutine Status
  4. Producer-Consumer Pattern
← Back to Lua Academy