Throughput vs latency trade-offs
Balance total work per second (throughput) vs time per item (latency): compare per-item processing, batching, and degree-of-parallelism tuning.
Throughput vs latency trade-offs is a free C# Academy lesson on CoddyKit — lesson 3 of 3. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the C# Academy learning path, one of 3 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Throughput vs latency
Definitions:
- Throughput: items per second
- Latency: time to finish one item
- Trade-off: batching and higher parallelism may raise throughput but delay individual items
Per-item style
Process each item as it arrives: minimal wait per item, but overhead repeats for every item.
using System;
using System.Diagnostics;
using System.Threading;
using System.Threading.Tasks;
public class Program
{
// Simulate small per-item cost
static void HandleItem(int x)
{
// Fixed overhead per item
Thread.SpinWait(20000); // tiny CPU work
}
public static void Main(string[] args)
{
int n = 200;
Stopwatch sw = Stopwatch.StartNew();
for (int i = 0; i < n; i++)
{
HandleItem(i); // process immediately (no batching)
// emit result right away (low latency style)
}
sw.Stop();
Console.WriteLine("Per-item style: {0} ms for {1} items", sw.ElapsedMilliseconds, n);
}
}
Batching style
Batching reduces repeated overhead and boosts throughput, but early items wait for the batch to fill (higher latency).
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Threading;
public class Program
{
static void ProcessBatch(List<int> batch)
{
// Amortize overhead across the whole batch
Thread.SpinWait(20000); // one-time overhead
for (int i = 0; i < batch.Count; i++)
{
// small per-record work
int val = batch[i] * 2;
if (val == int.MinValue) { } // keep compiler from dropping work
}
}
public static void Main(string[] args)
{
int n = 200;
int batchSize = 20;
List<int> current = new List<int>(batchSize);
Stopwatch sw = Stopwatch.StartNew();
for (int i = 0; i < n; i++)
{
current.Add(i);
if (current.Count == batchSize)
{
ProcessBatch(current);
current.Clear(); // emit results after the batch finishes
}
}
if (current.Count > 0) ProcessBatch(current);
sw.Stop();
Console.WriteLine("Batch style: {0} ms for {1} items (batch={2})", sw.ElapsedMilliseconds, n, batchSize);
}
}
Parallelism tuning
Tuning MaxDegreeOfParallelism can raise throughput for CPU-bound work; too high may hurt due to context switches.
using System;
using System.Diagnostics;
using System.Threading;
using System.Threading.Tasks;
public class Program
{
static void Work(int x)
{
// CPU-bound unit
Thread.SpinWait(40000);
}
public static void Main(string[] args)
{
int[] data = new int[200];
for (int i = 0; i < data.Length; i++) data[i] = i;
foreach (int dop in new int[] { 1, 2, 4 })
{
var opt = new ParallelOptions();
opt.MaxDegreeOfParallelism = dop;
Stopwatch sw = Stopwatch.StartNew();
Parallel.ForEach(data, opt, Work);
sw.Stop();
Console.WriteLine("DOP={0} -> {1} ms", dop, sw.ElapsedMilliseconds);
}
}
}
Micro-batch idea
Micro-batches can balance both goals: better throughput than per-item, lower latency than huge batches.
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Threading;
using System.Threading.Tasks;
public class Program
{
static void ProcessBatch(List<int> batch)
{
Thread.SpinWait(15000); // small shared overhead
for (int i = 0; i < batch.Count; i++) Thread.SpinWait(2000);
}
public static void Main(string[] args)
{
int n = 200;
int micro = 5; // micro-batch size
List<int> buf = new List<int>(micro);
Stopwatch sw = Stopwatch.StartNew();
for (int i = 0; i < n; i++)
{
buf.Add(i);
if (buf.Count == micro)
{
ProcessBatch(buf);
buf.Clear(); // emit more frequently than big batches
}
}
if (buf.Count > 0) ProcessBatch(buf);
sw.Stop();
Console.WriteLine("Micro-batch (size={0}): {1} ms", micro, sw.ElapsedMilliseconds);
}
}
Tuning tips
Tuning guide:
- Measure both ms/item and items/sec
- Try micro-batches first
- Increase parallelism slowly; watch CPU and context switches
- Bound queues to avoid long waits and memory growth
Throughput vs latency trade-off
Recap
Recap: Per-item = low latency, lower throughput. Big batches/high DOP = higher throughput, higher latency. Micro-batches and careful DOP tuning help balance both.
Frequently asked questions
Is the “Throughput vs latency trade-offs” lesson free?
Yes — the full text of “Throughput vs latency trade-offs” is free to read here on the web, and the C# Academy course includes 3 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the C# Academy course, upgrade to CoddyKit PRO.
What will I learn in “Throughput vs latency trade-offs”?
Balance total work per second (throughput) vs time per item (latency): compare per-item processing, batching, and degree-of-parallelism tuning. You practise C# Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start C# Academy?
No prior experience is required. C# Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 3, so you can start here or from the beginning and move at your own pace.
How long does the “Throughput vs latency trade-offs” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this C# Academy lesson?
Yes. Every C# Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Parallel.ForEach, PLINQ
- Producer/consumer with Channels (overview)
- Throughput vs latency trade-offs