The Data Reuse Problem

A kernel can be correct yet slow because it keeps fetching the same data from slow global memory over and over. 🐢

On the GPU, arithmetic is cheap but reaching global memory is expensive. Many kernels wait on memory far more than they compute.

Data reuse means one value loaded from global memory is used by many computations instead of being read again each time.

Picture blurring an image. Each output pixel averages its neighbors, so every input pixel gets read by several different threads.

In that blur, the value in[i] is read by threads i-1, i, and i+1. The same byte travels across the slow PCIe-fed bus three times.

With a wider window or a 2D grid, each element may be re-read dozens of times. This redundant traffic dominates the runtime.

Global memory has a fixed peak bandwidth. Reading the same data repeatedly wastes that budget on bytes you already had.

While warps stall waiting on repeated global loads, the math units sit idle. You paid for cores you are barely using. 😴

Arithmetic intensity is the ratio of math operations to bytes moved. Low intensity means memory, not compute, limits you.

The fix is to load each needed value once into fast on-chip storage, then let many threads reuse it from there.

That fast on-chip storage is shared memory. Staging data there is the foundation of every tiling optimization ahead.

Why is a naive stencil kernel often slow?

Naive kernels re-read shared data from global memory, wasting bandwidth and stalling compute. Tiling exists to load once and reuse. ✅