การแบ่งงานข้าม GPU

Two GPUs can finish a job in roughly half the time, but only if you split the work. The art is partitioning: deciding which GPU handles which part.

The classic strategy is to cut the data, not the code. With domain decomposition each GPU gets its own slice of the array or grid to process.

For a 1D array, just divide its length. Give the first chunk of elements to GPU 0 and the next chunk to GPU 1, and so on.

Every GPU needs the start of its slice. The offset for device d is simply d times the chunk size, marking where its data begins.

Each GPU needs its own buffer. Set the device, then cudaMalloc space just for that card's slice instead of the whole array.

Upload to each GPU only the portion it owns. Copy from host[offset] into that device's buffer so no card holds data it will not touch.

Loop over the GPUs, set each current, and launch the kernel on its slice. The launches are asynchronous, so all cards start working in parallel.

When kernels finish, copy each device's output back into the right spot of the host array using its offset. The pieces reassemble into one result.

If n does not divide evenly, the last GPU must handle the remainder. Give it the extra elements so nothing in the array is skipped.

Stencil and neighbor operations read across slice edges. Those halo regions must be shared between GPUs, or results at the borders go wrong.

If one GPU is faster, an even split wastes it. Good load balancing gives the stronger card a bigger slice so both finish at the same time.

Recall the standard way to spread one large dataset across several GPUs.

You split data into slices, allocate and copy per device, launch on each, then gather results. Mind the remainder and halos. Next: copying directly GPU to GPU. ✨