Создание и использование потоков выполнения

A non-default stream is your own work queue. Two streams run independently, so the GPU can make progress on both at once. 🧵

A stream lives in a cudaStream_t handle. You create it once, pass it to your calls, and destroy it when you are done.

Call cudaStreamCreate to make a fresh stream. It allocates the queue the driver uses to schedule that stream's work.

A kernel's launch config has a fourth argument: shared memory size, then the stream. Pass your stream to queue the kernel there.

Put one job on streamA and another on streamB. With no dependency between them, the GPU is free to run both concurrently.

Stream calls are asynchronous: the host returns immediately and keeps going. The GPU works through the queue on its own time.

Use cudaStreamSynchronize to block the host until just that stream finishes, instead of waiting on the whole device.

Want a peek? cudaStreamQuery returns cudaSuccess if the stream is done and cudaErrorNotReady if it is still busy, without stalling you.

When a stream is no longer needed, release it with cudaStreamDestroy. Pending work still completes before the handle is freed.

Create a stream with the cudaStreamNonBlocking flag so it does not implicitly synchronize with the legacy default stream.

Spreading independent chunks over several streams lets the scheduler keep the engines fed. Beyond a few, returns diminish as hardware fills up.

How do you queue a kernel into a specific stream?

You can now create streams, launch kernels in them, sync or query, and destroy them. Independent streams unlock real concurrency. ✨