Der PCIe-Transferengpass

The CPU and GPU live on separate boards connected by the PCIe bus. Every cudaMemcpy must squeeze through this narrow bridge. 🌉

GPU memory bandwidth can be terabytes per second, but PCIe delivers only tens of gigabytes. The bus is the slow link.

For a quick kernel, the transfer time can dwarf the compute time. Your GPU sits idle waiting for data to arrive.

The first fix is simple: move only what you need and do more work per byte once the data is on the GPU.

Avoid shuttling arrays back and forth. Keep them resident on the device across several kernels instead of re-uploading.

Many tiny transfers each pay a fixed overhead. Batching them into one big copy is far more efficient. 📦

You can hide transfer cost by overlapping copies with compute using streams, so the GPU works while data flows.

Pinned host memory enables faster DMA and async copies. It is the key to truly overlapping transfers with kernels.

Time your copies and kernels separately. A profiler like Nsight shows exactly where the bus is hurting you.

If you are transfer-bound, a faster kernel will not help. Cut data movement first, then optimize compute.

For tiny problems, the copy cost can outweigh the speedup. The GPU pays off when the compute clearly dwarfs the transfer.

Profiling shows your program spends most of its time moving data over PCIe. What helps most?

You saw why PCIe is the slow link: copy less, keep data resident, batch transfers, overlap with streams, and always measure. 🎉