メモリ階層のメンタルモデル

Think of GPU memory as a pyramid. The top is tiny and lightning fast, while the base is huge but slow. Your job is to use each layer well.

At the peak sit registers: per-thread, fastest, and very limited. Keep your hottest working values here whenever you possibly can.

Just below comes shared memory, an on-chip scratchpad visible to all threads in a block. It is your tool for fast intra-block teamwork.

Alongside sits the constant cache, ideal for small read-only values that a warp reads uniformly and broadcasts in one fetch.

The wide base is global memory: gigabytes, visible to everyone, but high-latency. It holds your big inputs and outputs.

Watch out for local memory. It sounds close by but lives in slow DRAM, used only when registers spill. Avoid relying on it.

Pick by who needs the data. One thread alone wants registers, a block working together wants shared memory, everyone wants global.

Registers and shared memory vanish when a kernel ends, but global memory persists across launches. Match storage to how long data must live.

The winning pattern is load once from global, compute in fast on-chip memory, then write once back. This minimizes slow global memory traffic.

Spending lots of registers or shared memory per block lets fewer blocks run at once. Occupancy is the balance you constantly tune.

Great kernels deliberately route each piece of data to the right layer. That single habit is what separates slow code from fast CUDA code. 💪

Two threads in the SAME block need to share intermediate results. Which space fits best?

You built a mental map: registers, shared, constant, and global trade speed for size and scope. Routing data to the right layer is the whole game. 🧠