0Pricing
Go Academy · Lesson

Pipeline and Stage Patterns

Composing multi-stage concurrent pipelines

Pipeline and Stage Patterns is a free Go Academy lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Go Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

What is a pipeline?

A concurrent pipeline is a series of stages connected by channels. Each stage receives values from upstream, transforms them, and sends results downstream, allowing stages to run concurrently.

Simple three-stage pipeline

Generate → transform → consume:

func generate(nums ...int) <-chan int {
    out := make(chan int)
    go func() { defer close(out); for _, n := range nums { out <- n } }()
    return out
}
func square(in <-chan int) <-chan int {
    out := make(chan int)
    go func() { defer close(out); for n := range in { out <- n*n } }()
    return out
}

for v := range square(generate(2, 3, 4)) {
    fmt.Println(v) // 4, 9, 16
}

Stage pattern

Each stage is a function that takes an input channel and returns an output channel. Stages compose cleanly and can be parallelised independently.

Fan-out within a pipeline

Fork a stage into N goroutines to parallelise CPU-bound work:

func parallelSquare(in <-chan int, n int) <-chan int {
    outs := make([]<-chan int, n)
    for i := 0; i < n; i++ {
        outs[i] = square(in) // all read from same in
    }
    return merge(outs...)
}

Cancellation with context

Pass a context to each stage. When it is cancelled, stages stop reading and close their output channels, draining the entire pipeline.

func stage(ctx context.Context, in <-chan int) <-chan int {
    out := make(chan int)
    go func() {
        defer close(out)
        for {
            select {
            case v, ok := <-in:
                if !ok { return }
                select {
                case out <- process(v):
                case <-ctx.Done(): return
                }
            case <-ctx.Done(): return
            }
        }
    }()
    return out
}

Back-pressure

Use buffered channels between stages to decouple producer and consumer speed. Without buffering, a slow stage stalls the entire pipeline.

out := make(chan Result, 50) // buffer 50 results

Error propagation

Wrap values in a result struct to propagate errors through the pipeline without panicking:

type item struct { val int; err error }

Done channel pattern

Use a done channel as an alternative to context for simpler pipelines. Close done to signal all stages to stop.

done := make(chan struct{})
defer close(done)

Ordering guarantees

A simple pipeline preserves order if each stage is sequential. Parallelised stages do not preserve order — use an index field to re-order results downstream if needed.

Pipeline with WaitGroup

The last stage does not need to close a channel — just range over it. Use WaitGroup in fan-in to close the merged channel after all producer goroutines finish.

Real-world use

Common pipeline applications: ETL data processing, image resizing, log parsing, build systems, and streaming request handlers.

Quick Check

How do you propagate cancellation through a multi-stage pipeline?

Recap: Pipeline Pattern

Key points:

  • Stage: func(in <-chan T) <-chan U — compose for pipelines
  • Cancellation via context; back-pressure via buffered channels
  • Fan-out within a stage for parallelism
  • Wrap values in result struct for error propagation

Frequently asked questions

Is the “Pipeline and Stage Patterns” lesson free?

Yes — the full text of “Pipeline and Stage Patterns” is free to read here on the web, and the Go Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Go Academy course, upgrade to CoddyKit PRO.

What will I learn in “Pipeline and Stage Patterns”?

Composing multi-stage concurrent pipelines You practise Go Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Go Academy?

No prior experience is required. Go Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Pipeline and Stage Patterns” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Go Academy lesson?

Yes. Every Go Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Pipeline and Stage Patterns
  2. errgroup for Concurrent Error Handling
  3. Semaphore Pattern
  4. Leak Detection and Cancellation
← Back to Go Academy