cudaMemcpyAsync в потоке выполнения

Plain cudaMemcpy stops your CPU until the copy finishes. Your host thread just waits, doing nothing useful while bytes move.

Meet cudaMemcpyAsync. It queues the copy and returns to your CPU immediately, so the host can keep working while the transfer happens.

The async copy takes an extra argument: a stream. The stream is the ordered queue where the transfer waits its turn to run.

Here is the catch: async copies only run truly non-blocking from pinned host memory. Pass a pageable buffer and CUDA silently falls back to a blocking copy.

When the call returns, the copy has only been scheduled, not completed. The data may still be in flight, so do not read it yet.

Work in one stream runs in order, one item after another. So a copy issued before a kernel in the same stream is guaranteed to finish first.

Before touching the results on the CPU, make sure the queue drained. Call cudaStreamSynchronize to block until that stream's work is done.

Async copies let a transfer in one stream run while a kernel in another stream computes. That concurrency is how you hide transfer time.

If you pass stream 0, the legacy default, it can serialize against everything else. Use your own created streams to actually get overlap.

Since the copy finishes later, the host buffer must stay valid until then. Free it too early and the in-flight transfer reads garbage.

The classic flow is async copy in, launch the kernel, then async copy out, all in one stream. Sync only at the very end.

What does cudaMemcpyAsync require to be genuinely non-blocking?

cudaMemcpyAsync queues a copy in a stream and returns at once, but needs pinned memory to overlap. Sync the stream before reading results. ⚡