Watermarks

How to decide a time window is "complete" when events arrive out of order and we can never be absolutely certain no more will come.

Problem

In an event-time stream, reception order does not reflect origin order: an AS4 message delayed by a broken access point may arrive 10 minutes after a message emitted 5 seconds later elsewhere. When to close a window [09:00, 10:00]? Wait too long and latency becomes unacceptable. Close too early and data is lost. The watermark mechanism resolves this dilemma by providing a continuous estimate of "how far we've seen everything there was to see".

Forces

• Heuristic vs perfection — there is no perfect watermark for real distributed flows.
• Latency/completeness trade-off: conservative watermark = latency ↑, completeness ↑.
• Source-aware: Kafka offers a natural watermark (per-partition timestamps); HTTP push does not.
• Skew per key: different partners have different typical delays.
• Failures: a dead consumer stalls the watermark on its partition.

Solution

A watermark W(t) is an estimate: "all events with timestamp ≤ W(t) have normally arrived at time t". Common strategies: strict (W = max(ts) - X minutes — Flink BoundedOutOfOrderness), percentile (W = 99th percentile of recent arrivals), per-source (each Kafka partition emits its own watermark, the system takes the min). When W(t) passes the end of a window, the window closes and the aggregate is emitted. Later events (late data) are handled by allowed lateness (re-emit updates) or side output (send to a dedicated queue for reprocessing). Google Dataflow and Apache Beam expose a WindowFn.assignWindows + Trigger API to decouple these concerns.

Structure

Event stream (event-time, processing-time):
  (09:00, 09:01)  ●
  (09:02, 09:02)  ●
  (09:05, 09:08)  ●  ◄── 3 min late
  (09:06, 09:06)  ●
  (08:58, 09:10)  ●  ◄── 12 min late (very out of order)
  (09:10, 09:11)  ●

Window [09:00, 09:05)
  Strategy "max(ts) - 2min":
    At PT=09:08, max event-time seen = 09:05
    Watermark = 09:05 - 2min = 09:03
    Window not yet closed (watermark < 09:05)
  At PT=09:11, max event-time seen = 09:10
    Watermark = 09:10 - 2min = 09:08
    Window CLOSED, emit aggregate of {09:00, 09:02}
    Late event 08:58 arriving at 09:10:
      → drop, or side-output to late_events topic, or
      → if allowed lateness 5min: update aggregate to include 08:58

EDI implementation

In EDI, source-aware watermarking is almost always the right approach. (1) Multi-access-point PEPPOL hub: each AP has its own cadence, compute one watermark per source AP and global watermark = min. If an AP is down, its watermark stalls and blocks downstream windows — by design, because we do not want to close a DRR report without data from all APs. (2) Fiscal reporting (DRR, ZATCA, CFDI): conservative watermark with minimum 24h lag to absorb partner outages; report emitted at D+1 with 7-day allowed lateness to allow slow re-deliveries. (3) Anomaly detection: aggressive watermark (30s lag) to react fast; late events go to a side output for batch reprocessing. Never use processing-time for these critical cases — a replay produces different watermarks than the original run.

Anti-patterns

• Processing-time watermark for fiscal reporting — replay inconsistency guaranteed.
• Single global watermark (max over all partitions) — a dead consumer stalls it for the whole job.
• Lag too short — windows close prematurely, data lost silently.
• No side output on late data — silently lost, possible fiscal under-reporting.
• Unlimited allowed lateness — RocksDB state never released.

Related patterns

• Tumbling Window — what watermarks close.
• Sliding Window — same.
• Session Window — same.
• Exactly-Once Semantics — often combined with watermarks for exact reporting.

Sources

• Akidau T. et al. — The Dataflow Model, VLDB 2015. The canonical source on watermarks. research.google
• Akidau T. — The world beyond batch: Streaming 102, O'Reilly 2015. oreilly.com
• Apache Flink — Generating Watermarks. nightlies.apache.org
• Apache Beam — Programming Guide: Watermarks and late data. beam.apache.org
• Akidau, Chernyak, Lax — Streaming Systems, O'Reilly 2018, ch. 2-3.