กับดักของสตรีมเริ่มต้น

A stream is an ordered queue of GPU work. Operations in the same stream run one after another, in the order you issued them. ⏳

If you never name a stream, every call lands in the default stream, also called stream 0. It is where all your work has secretly been running.

The default stream is synchronizing: it blocks until other streams finish, and they wait for it. Nothing overlaps, so your GPU sits idle between steps.

Issue a copy, a kernel, then another copy on stream 0 and they run strictly back to back. The GPU can never start step two before step one ends.

Modern GPUs have separate engines for compute and for copying. On the default stream those engines take turns instead of working together.

Data transfers over PCIe are slow. When copies cannot overlap with compute, that transfer time is added directly onto your total runtime.

Many default-stream calls are blocking on the host too. cudaMemcpy returns only after the copy is done, stalling your CPU.

By default, work in any stream waits for stream 0, and stream 0 waits for them. This legacy rule quietly destroys concurrency you thought you had.

In a profiler the default-stream trap shows up as a single busy lane with gaps, while copy and compute engines sit idle waiting for each other.

The fix is to issue independent work on separate streams. Then copies and kernels can run at the same time and fill those idle gaps.

Compiling with --default-stream per-thread gives each host thread its own default stream, easing the trap without rewriting every call.

Why does putting all work on the default stream hurt performance?

The default stream serializes your GPU work and blocks overlap. To go faster, you will move independent tasks onto their own streams next. 🚀