Throughput vs latency trade-offs
Balance total work per second (throughput) vs time per item (latency): compare per-item processing, batching, and degree-of-parallelism tuning.
Throughput vs latency
Definitions:
- Throughput: items per second
- Latency: time to finish one item
- Trade-off: batching and higher parallelism may raise throughput but delay individual items
Per-item style
Process each item as it arrives: minimal wait per item, but overhead repeats for every item.
using System;
using System.Diagnostics;
using System.Threading;
using System.Threading.Tasks;
public class Program
{
// Simulate small per-item cost
static void HandleItem(int x)
{
// Fixed overhead per item
Thread.SpinWait(20000); // tiny CPU work
}
public static void Main(string[] args)
{
int n = 200;
Stopwatch sw = Stopwatch.StartNew();
for (int i = 0; i < n; i++)
{
HandleItem(i); // process immediately (no batching)
// emit result right away (low latency style)
}
sw.Stop();
Console.WriteLine("Per-item style: {0} ms for {1} items", sw.ElapsedMilliseconds, n);
}
}
All lessons in this course
- Parallel.ForEach, PLINQ
- Producer/consumer with Channels (overview)
- Throughput vs latency trade-offs