ediverse

Type to search across standards, messages, tools and glossary.

ediverse Explore the platform

Spotlight PEPPOL BIS Billing 3.0 The EU e-invoicing mandate is here — France Sept 2026, Belgium Jan 2026, Germany 2025.

— 01

Foundations

— 02

Standards

— 03

Flagship messages

— 04

Tools

Idempotency and deduplication

The same order must never be delivered twice. The pattern is easy to state, brutal to fix when a retry storm has injected 14,000 duplicates into the ERP.

Problem

In every real EDI exchange, messages can be replayed. The classic case: the partner received and processed the message, but the MDN was lost on the return network. The sender, following its SLA, retransmits. Without idempotency on the receiver side the order is integrated twice — and each duplicate costs business time to isolate and reverse (double receipt, double invoice, possibly double payment).

Forces

  • The sender doesn't know for sure. If the MDN is lost, the sender cannot tell "partner didn't receive it" from "partner received it but the MDN was lost". Consequence: it must retry as a safety net.
  • The receiver must tolerate duplicates. It carries the dedup responsibility because it is the only side with a consistent view of what it has already processed.
  • The dedup window has a cost. Keeping the full history of every received reference is expensive in storage and lookup unless bounded by a window.
  • EDI standards provide the key. UNB, ISA, AS4 each define per-interchange unique references — provided partners use them correctly and agree on the window.

Solution: one uniqueness key per send

The Idempotent Receiver pattern (Hohpe & Woolf, 2003) prescribes:

  1. Guarantee on the sender side that a retransmission uses strictly the same uniqueness key as the original send — not a new one.
  2. On the receiver side, hold a table of received keys over a known time window (typically 30, 90 or 180 days in EDI). On receipt, check absence; if the key exists, the message is a duplicate — return the original ACK (to reassure the sender) but do not integrate.
  3. Define an explicit behaviour outside the window: in practice, treat the message as new and integrate it (with a warning log).

EDIFACT — UNB Interchange Control Reference

The UNB (Interchange Header) segment carries an Interchange Control Reference (composite 0020) that must be unique between the two partners over a negotiated window. ISO 9735 does not fix that window — it is left to the bilateral agreement. European retail (EANCOM) recommends 14 alphanumeric characters unique over 90 days. The reference is echoed in the UNZ trailer for bracketing.

edifact interchange-with-ctrl.edi 
UNB+UNOC:3+SENDER:14+RECEIVER:14+260514:1545+ORD202605140001'
UNH+1+ORDERS:D:96A:UN:EAN008'
...
UNZ+1+ORD202605140001'

Reading: ORD202605140001 above follows a Walmart-style convention — business prefix ORD, date 20260514, counter 0001. This shape ensures no accidental retransmission will appear under a new reference.

X12 — ISA13 and GS06

In X12 the idempotency key is composite. ISA13 (Interchange Control Number) is the interchange control number — nine incrementing digits, unique per sender. GS06 (Group Control Number) is the functional group number, unique within an interchange and echoed in GE02. For deduplication, the pair (sender, ISA13) is the canonical primary key.

plaintext x12-with-isa13.edi 
ISA*00*          *00*          *ZZ*SENDERID       *ZZ*RECEIVERID     *260514*1545*U*00401*900042001*0*P*>~
GS*PO*SENDERID*RECEIVERID*20260514*1545*42*X*004010~
ST*850*0001~
...
SE*42*0001~
GE*1*42~
IEA*1*900042001~

The X12 envelope stores ISA13 over 9 characters, so a rollover window of 109 sends — at 100,000 sends per day, ~27 years before a theoretical collision. More than enough for a 6-month contractual window in practice.

AS4 — eb:MessageId and duplicate detection

AS4 standardises dedup at the protocol level. The eb:Messaging block carries a mandatory eb:MessageId — typically a UUID, unique by construction. The AS4 specification (§3.2.3.3) defines an optional Duplicate Detection feature the receiver can enable: it logs the MessageId and, on receipt of a duplicate, re-emits the same Receipt as the first time without re-delivering the payload to the application.

In the PEPPOL network, duplicate detection is mandatory on every Access Point. That is what lets a sender retry without fear after a timeout: the guarantee is carried by the network, not by the business application.

Receiver-side uniqueness window

Three questions to ask explicitly:

  • Window duration. 30 days is the minimum, 90 days the recommended value for operational flows (orders, despatch advices), 180 days to 1 year for financial flows (invoices, payments) where replays can occur much later.
  • Key granularity. The key can be (a) the envelope reference alone (UNB CTRL, ISA13), (b) the envelope reference + sender identifier, or (c) the reference + payload hash. Option (b) is the standard; option (c) also protects against malicious replays.
  • Behaviour on duplicate. Re-emit the same ACK as the first time — that reassures the sender without polluting the business log. Never return a "duplicate" error that could trigger spurious escalations.

Anti-patterns

  • New key on every retransmission. A sender that generates a different UNB CTRL on each attempt breaks deduplication entirely — the receiver integrates the order N times. Always reuse the initial key on retries.
  • Infinite window. Keeping every received reference forever is useless and costly. Always set an explicit window and purge beyond it.
  • Idempotency on the sender side only. Believing the sender alone can prevent duplicates is naive — it does not know whether the receiver actually integrated. The pattern must be implemented on the receiver side.
  • Dedup by payload hash only. If two genuinely distinct orders happen to have identical content (same customer, same product, same quantity, same date), the hash matches and the second is rejected by mistake. Always use the envelope reference as primary key, the hash only as additional check.
  • Resetting ISA13 counters. A partner that resets ISA13 to 1 after a restart breaks uniqueness — counter must persist in a database, not in memory.

Sources