CPU vs. GPU: Latenz vs. Durchsatz

A CPU and a GPU both run code, but they are built for opposite goals. One races to finish a single task; the other floods a wall of work all at once. 🏎️

Latency is the time it takes to complete one single task from start to end. When you want one answer right now, low latency is what you care about.

Throughput is how many tasks you complete per second in total. You can have high throughput even if each single task is a bit slow.

A CPU has a few powerful cores with huge caches and clever tricks to rush one task to the finish line as fast as possible.

A GPU trades single-task speed for sheer volume: thousands of small cores chew through enormous batches of work at the same time.

Think of a CPU as a handful of strong workers and a GPU as an army of thousands of simple ones. Different shapes for different jobs.

A sports car is low latency: one trip is quick. A bus is high throughput: each trip is slower, but it moves far more people per hour. 🚌

CPUs spend transistors on caches and branch prediction. GPUs spend them on raw arithmetic units, so more of the chip is doing actual math.

A GPU does not make each task fast; it hides slow memory waits by instantly switching to other ready work. Idle time gets filled, not removed.

For one quick decision, branchy logic, or a task that cannot split up, the low-latency CPU is still the right tool. Volume is not everything.

The moment you have the same operation to run over thousands of items, the GPU's massive parallelism turns a long job into a quick one.

Let us pin down the core tradeoff between these two chips.

CPUs use a few big cores for low latency; GPUs use thousands of tiny cores for high throughput. Pick the chip that matches the shape of your work. 👍