ediverse

Type to search across standards, messages, tools and glossary.

ediverse Explore the platform

Spotlight PEPPOL BIS Billing 3.0 The EU e-invoicing mandate is here — France Sept 2026, Belgium Jan 2026, Germany 2025.

— 01

Foundations

— 02

Standards

— 03

Flagship messages

— 04

Tools

Back-pressure

When downstream cannot keep up, slow upstream — rather than letting queues swell until memory is saturated or messages are lost.

Problem

An EDI parser ingests 5,000 INVOIC/s; the downstream semantic validator only handles 2,000. Without a throttling mechanism, the in-between queue grows indefinitely. Three possible outcomes, all bad: (a) OOM kill of the broker, (b) latency drifting to hours, (c) silent drops when the buffer is full.

Forces

  • Capacities are unequal. Validator, enricher, ERP writer have different throughputs.
  • Spikes happen. A partner replaying 24h of DESADV after an outage emits 100× the normal rate.
  • Silent drops are forbidden in regulatory EDI.
  • Slowdown must cascade upstream. If the validator slows, the parser must slow, and the network ingress must reject new connections.

Solution

Put in place a saturation signal propagated from consumer to producer: at each stage, the receiver advertises its capacity (TCP window size, Reactive Streams demand, Kafka drain timer), and the sender self-regulates. When pressure rises, the sender pauses its upstream producer, which pauses its own, all the way up to the network ingress that refuses new connections (HTTP 429 / 503 with Retry-After). 

producer ──messages──► [queue] ──messages──► consumer
                          │
                          ▼  capacity 10 000
   producer ─◄─ "slow down" / 429 / TCP RWIN=0 ─◄─ broker detects 95% full
   producer pauses or rejects new work

EDI implementation

In EDI, back-pressure materialises at several layers: (1) AS2/AS4 responds 503 Service Unavailable with Retry-After: 300 when the hub is saturated; (2) Kafka consumer groups detect a lag > 30s on a topic and throttle the upstream producer via a control loop; (3) RabbitMQ activates native flow control beyond 95% memory used; (4) Reactive Streams (RxJava, Project Reactor, Akka Streams) implement the pattern in application code with a request(n) that explicitly asks the upstream for N elements.

Anti-patterns

  • Unbounded queue. No back-pressure: the broker absorbs until OOM kill.
  • Silent drop when full. Unobserved loss, very expensive in EDI.
  • Sleep on the producer side. Blocking the thread rather than explicitly signalling consumer-side pressure. Works briefly, does not monitor.
  • No cross-stage propagation. Stage 4 slows but stage 1 keeps pumping: the intermediate queue explodes.
  • Rate Limiter — fixed-rate throttling on the sender side, complementary.
  • Bulkhead — pool isolation to prevent saturation propagation.
  • Circuit Breaker — abrupt cut-off when downstream fails rather than slows.

Sources