Partitioning Work Across GPUs
Domain decomposition strategies.
Many GPUs, One Job
Two GPUs can finish a job in roughly half the time, but only if you split the work. The art is partitioning: deciding which GPU handles which part.
Domain Decomposition
The classic strategy is to cut the data, not the code. With domain decomposition each GPU gets its own slice of the array or grid to process.
All lessons in this course
- Enumerating and Selecting Devices
- Partitioning Work Across GPUs
- Peer-to-Peer Memory Access
- Multi-GPU with NCCL