A Mental Model of the Hierarchy
Matching data to the right space.
A Mental Model of the Hierarchy is a free CUDA Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the CUDA Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
A Pyramid of Tradeoffs
Think of GPU memory as a pyramid. The top is tiny and lightning fast, while the base is huge but slow. Your job is to use each layer well.
Top: Registers
At the peak sit registers: per-thread, fastest, and very limited. Keep your hottest working values here whenever you possibly can.
Next: Shared Memory
Just below comes shared memory, an on-chip scratchpad visible to all threads in a block. It is your tool for fast intra-block teamwork.
Side Path: Constant Cache
Alongside sits the constant cache, ideal for small read-only values that a warp reads uniformly and broadcasts in one fetch.
Base: Global Memory
The wide base is global memory: gigabytes, visible to everyone, but high-latency. It holds your big inputs and outputs.
The Trap: Local Memory
Watch out for local memory. It sounds close by but lives in slow DRAM, used only when registers spill. Avoid relying on it.
Scope Decides the Space
Pick by who needs the data. One thread alone wants registers, a block working together wants shared memory, everyone wants global.
Lifetime Decides Too
Registers and shared memory vanish when a kernel ends, but global memory persists across launches. Match storage to how long data must live.
The Golden Rule
The winning pattern is load once from global, compute in fast on-chip memory, then write once back. This minimizes slow global memory traffic.
Capacity Costs Occupancy
Spending lots of registers or shared memory per block lets fewer blocks run at once. Occupancy is the balance you constantly tune.
Putting It Together
Great kernels deliberately route each piece of data to the right layer. That single habit is what separates slow code from fast CUDA code. 💪
Quick Check
Two threads in the SAME block need to share intermediate results. Which space fits best?
Recap: Match Data to Layer
You built a mental map: registers, shared, constant, and global trade speed for size and scope. Routing data to the right layer is the whole game. 🧠
Frequently asked questions
Is the “A Mental Model of the Hierarchy” lesson free?
Yes — the full text of “A Mental Model of the Hierarchy” is free to read here on the web, and the CUDA Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the CUDA Academy course, upgrade to CoddyKit PRO.
What will I learn in “A Mental Model of the Hierarchy”?
Matching data to the right space. You practise CUDA Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start CUDA Academy?
No prior experience is required. CUDA Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “A Mental Model of the Hierarchy” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this CUDA Academy lesson?
Yes. Every CUDA Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Registers and Local Memory
- Global Memory Tradeoffs
- Constant Memory and Its Cache
- A Mental Model of the Hierarchy