Running a Pipeline

So far the pipeline has been a pure blueprint. Materialization is the process that turns that blueprint into running actors that actually move data.

Nothing happens until you explicitly run the graph, which is what makes Akka Streams composable and reusable.

Materialization requires an ActorSystem, which provides the threads and dispatcher that back the stream's stages. In modern Akka, the system also acts as the implicit materializer.

One ActorSystem typically serves an entire application and many concurrent streams.

The most direct way to run a Source is runWith, which attaches a Sink and materializes in one step, returning that Sink's materialized value.

Here the result is a Future[Int] that completes with the sum when the stream finishes.

If you have already built a closed RunnableGraph with to or toMat, call run() to materialize it. The return value is whatever materialized value the graph kept.

This separates pipeline construction from execution cleanly.

Sources offer shortcuts: runForeach, runFold, and runReduce each attach the corresponding Sink and run immediately.

They are concise for common terminal operations on a Source.

Terminal sinks return a Future that completes when the stream ends or fails. Register callbacks with onComplete to react to success or error.

Use the ActorSystem's dispatcher as the implicit ExecutionContext for these callbacks.

A typical data pipeline reads from a Source, transforms with Flows, performs async I/O with mapAsync, batches with grouped, and writes to a Sink.

Each stage is small and the whole thing materializes with a single run.

For resilience, wrap a Source or Flow with RestartSource.withBackoff so transient failures (a dropped connection) trigger an automatic restart with exponential backoff.

This keeps long-running ingestion pipelines alive without manual supervision.

A KillSwitch lets external code stop a running stream cleanly. Insert KillSwitches.single via viaMat and keep its materialized value to call shutdown() later.

This is essential for long-lived streams that must stop on application shutdown.

When the application exits, terminate the ActorSystem to free its threads. Chain the terminate call after the stream's completion Future so shutdown is orderly.

Leaking an ActorSystem keeps the JVM alive and holds resources open.

Materializing the same blueprint multiple times creates independent running streams that share the ActorSystem's resources. The blueprint itself stays immutable and side-effect free.

This makes it safe to define a pipeline once and run it on demand for each incoming job.

Consider what is required for a stream to actually process elements.

Running a pipeline means materializing a blueprint with an ActorSystem via run, runWith, or convenience operators, each returning a Future result.

You saw realistic multi-stage pipelines, automatic restarts with backoff, graceful shutdown via KillSwitch, resource cleanup with system.terminate(), and safe reuse of an immutable blueprint across independent runs.