追踪数据的采样策略

Capturing every trace in a busy system produces enormous data volumes. Sampling keeps a representative subset to control storage and processing cost while preserving useful insight.

A service handling thousands of requests per second can emit millions of spans per minute. Storing all of them is expensive and rarely necessary for diagnosis.

• Network overhead
• Backend storage
• Query latency

Head-based sampling decides at the start of a trace whether to keep it, before the outcome is known. It is cheap and simple.

A common head-based form keeps a fixed percentage. The decision is made on the trace ID so all spans in a trace agree.

Tail-based sampling waits until a trace finishes, then decides using the full picture. It can prioritize errors and slow requests.

Each approach has costs.

• Head-based: cheap, but may drop the rare error you needed
• Tail-based: keeps interesting traces, but buffers spans and uses more memory

The sampling decision must be consistent across services so a trace is kept whole, not in fragments. The decision propagates via the trace context.

Rate-limiting samplers cap traces per second, protecting the backend during traffic spikes regardless of percentage.

The OpenTelemetry Collector can apply tail sampling centrally, freeing apps from the decision.

Start with head-based probabilistic sampling for simplicity. Move to tail-based when you must guarantee that errors and slow traces are always captured.

Combine strategies: sample normal traffic lightly but keep 100% of errors and high-latency traces. This maximizes signal per stored byte.

Pick the strategy that guarantees error traces are kept.

You learned why traces are sampled, how head-based sampling decides up front cheaply while tail-based waits for the full trace to keep errors and slow requests, and that decisions must propagate consistently. Combining light sampling of normal traffic with full capture of important traces gives the best signal for the cost.