ストリーム処理における時間の意味論

In stream processing, when an event happened is often more important than when you processed it.

Choosing the wrong notion of time leads to incorrect counts, broken windows, and misleading analytics.

Event time is the timestamp embedded in the event itself — when it actually occurred at the source.

• A purchase made at 14:03 carries 14:03 regardless of network delays.
• It produces deterministic, replayable results.

Processing time is the wall-clock time of the machine running the stream operator when it sees the event.

• Simple and low-latency.
• But non-deterministic — the same data reprocessed later yields different windows.

Ingestion time is when the event entered the streaming system (e.g., appended to a Kafka topic).

It is a middle ground: more stable than processing time, but still not the true moment the event occurred.

For one event, the order is usually:

• Event time (created at source)
• then ingestion time (arrives in the system)
• then processing time (operator reads it)

The gaps between them are caused by network and queueing delays.

Events rarely arrive in event-time order. A mobile device offline for an hour may deliver events long after they occurred.

If you window by event time, the engine must wait for and correctly slot these late arrivals.

A watermark is the engine's estimate that no events older than time T will still arrive.

• It lets the system decide when an event-time window is complete.
• It trades latency for completeness — wait longer, catch more late events.

To use event time, you tell the engine how to read the timestamp from each record's payload.

Pick based on requirements:

• Event time — analytics, billing, anything needing correctness and replayability.
• Processing time — real-time monitoring where approximate is fine.
• Ingestion time — when source timestamps are unreliable.

Most engines let you configure allowed lateness — a grace period after the watermark during which late events still update results.

Events arriving after that are dropped or routed to a side output for separate handling.

Correct time handling means:

• Carry an event-time timestamp in every record.
• Use watermarks to know when windows are done.
• Set allowed lateness for stragglers.
• Prefer event time for any result that must be reproducible.

Test your understanding of time semantics.

You learned the three core time semantics.

• Event time = when it happened; processing time = when read; ingestion time = when it entered the system.
• Watermarks decide when event-time windows are complete.
• Allowed lateness handles stragglers.
• Use event time for correctness and replayability.