Message Deduplication Ledger

When 12 microservices in a hub ingest AS2, AS4, SFTP, REST — each with its own dedup cache — you end up with 12 diverging sources of truth. A shared centralised ledger fixes that and provides a cross-flow audit view.

Problem

A typical 2026 multi-tenant EDI hub ingests from 6 channels (AS2, AS4, OFTP2, SFTP, PEPPOL, REST API) routed through 8-12 internal microservices (EDIFACT parser, X12 parser, UBL normaliser, XSD/Schematron validator, mapping engine, publish to Kafka, etc.). Each microservice must dedup what it receives — otherwise a replayed message at ingestion creates a cascade of duplicates downstream. Maintaining 12 dedup caches per microservice is unmanageable: purge windows diverge, schemas differ, cross-flow audit is impossible ("has this message already been processed elsewhere in the hub?").

Forces

• Cross-flow consistency: one dedup decision shared by all hub components.
• Performance: O(1) expected lookup — typically Redis Cluster for hot path, PostgreSQL for audit persistence.
• Capacity: for a 100M messages/month hub, the ledger must handle ~3M reads/s, ~400 writes/s.
• Availability: the ledger becomes a SPOF — must be HA (Redis Sentinel/Cluster, PostgreSQL streaming replication).
• Retention window: flow-dependent. AS2 = 7d, AS4 PEPPOL = 30d, X12 = 60d, audit archive = 7-10y.

Solution

Define a single API dedup.checkAndRegister(scope, key, payload_hash) consumed by every hub component. The ledger combines two tiers:

• Hot tier (Redis): SETNX scope:key with TTL aligned to dedup window. Returns NEW or SEEN. p99 latency < 1ms.
• Cold tier (monthly partitioned PostgreSQL): same key persisted with detailed timestamps, payload hash, source service, scope. p99 latency ~5ms. Auditable source of truth.
• Sync write: Redis and PostgreSQL in write-through (Redis first, fallback PostgreSQL if Redis down). Eventual consistency acceptable because dedup is tolerant: a false negative (message seen but Redis says NEW) creates a duplicate → downstream consumer has an idempotent receiver.

Structure

                  ┌── EDI Hub ──────────────────────┐
                  │                                  │
   AS4 ingestion ─┼─► dedupClient.check("as4", id) ─┼─► HIT/MISS
   AS2 ingestion ─┼─► dedupClient.check("as2", id) ─┼─►
   PEPPOL inbound ┼─► dedupClient.check("peppol", id)┤
   SFTP poll      ─┼─► dedupClient.check("sftp", id) ┤
   EDIFACT Parser ┼─► dedupClient.check("edifact",ref)┤
   UBL Validator  ─┼─► dedupClient.check("ubl", uuid) ┤
                  │                                  │
                  └──────────┬───────────────────────┘
                             ▼
              ┌──── Deduplication Ledger ────┐
              │                               │
              │  Redis Cluster (hot, 30d TTL) │
              │           │                   │
              │           ▼ writes            │
              │  PostgreSQL (cold, 7y, partitioned)
              │           │                   │
              │           ▼ async S3 archive  │
              │  S3 Glacier (deep archive, fiscal audit)
              └───────────────────────────────┘

EDI implementation

// Client API (Node.js / TypeScript)
interface DedupResult {
  status: 'NEW' | 'SEEN';
  firstSeenAt?: string;       // ISO 8601
  payloadHashDrift?: boolean; // true if hash differs
  retryCount: number;
}

async function checkAndRegister(
  scope: string,         // "as4", "edifact-invoic", "peppol-ubl", ...
  key: string,           // AS4 MessageId, UNB 0020, BT-1 UBL, ...
  payloadSha256: string,
  ttlSeconds = 86400 * 30  // 30 days default
): Promise<DedupResult> {
  // Hot path Redis
  const redisKey = "dedup:" + scope + ":" + key;
  const setResult = await redis.set(
    redisKey, payloadSha256, 'NX', 'EX', ttlSeconds
  );

  if (setResult === 'OK') {
    // NEW: async insert into cold tier
    queueColdInsert({ scope, key, payloadSha256, firstSeenAt: now() });
    return { status: 'NEW', retryCount: 0 };
  }

  // SEEN: lookup cold for detailed stats
  const existing = await pg.query(
    'SELECT first_seen_at, payload_sha256, retry_count
     FROM dedup_ledger WHERE scope = $1 AND key = $2',
    [scope, key]
  );

  await pg.query(
    'UPDATE dedup_ledger SET retry_count = retry_count + 1
     WHERE scope = $1 AND key = $2',
    [scope, key]
  );

  return {
    status: 'SEEN',
    firstSeenAt: existing.rows[0].first_seen_at,
    payloadHashDrift: existing.rows[0].payload_sha256 !== payloadSha256,
    retryCount: existing.rows[0].retry_count + 1
  };
}

-- Monthly-partitioned PostgreSQL schema
CREATE TABLE dedup_ledger (
  scope            VARCHAR(40) NOT NULL,
  key              VARCHAR(200) NOT NULL,
  payload_sha256   CHAR(64) NOT NULL,
  source_service   VARCHAR(40),
  first_seen_at    TIMESTAMPTZ NOT NULL,
  last_seen_at     TIMESTAMPTZ DEFAULT now(),
  retry_count      INT DEFAULT 0,
  PRIMARY KEY (scope, key, first_seen_at)
) PARTITION BY RANGE (first_seen_at);

Multi-tenant edge case: prefix every scope with tenant_id to avoid cross-tenant collisions. scope = "tenant42:as4". The ledger should also expose an operational view ("did this message transit through the hub in the last 30 days?") via a (scope, key) or payload_sha256 search.

Anti-patterns

• In-process bloom filter — false positives (rejects valid messages), loss on crash, not auditable.
• No TTL — Redis memory blows up, ends up in silent LRU evictions.
• Ledger without cold archive — old messages get purged, fiscal auditability lost.
• Synchronous only to PostgreSQL — p99 latency ~50ms kills hub throughput.
• No monitoring on NEW/SEEN ratio — a sudden SEEN spike signals a partner in retry loop that must be investigated urgently.

Related patterns

• Idempotent Receiver for EDI flows — uses the ledger as backend.
• Transactional Inbox — complementary pattern at the broker level.
• Idempotency and deduplication — family view.
• Message Store — store everything, dedup later (alternative pattern).

Sources

• Helland P. — Idempotence Is Not a Medical Condition, ACM Queue, vol. 10 n° 4 (2012). queue.acm.org/detail.cfm?id=2187821
• Bloom B.H. — Space/Time Trade-offs in Hash Coding with Allowable Errors, CACM 1970. The founding Bloom filters paper — useful to understand why not to use them here.
• AWS — Amazon SQS FIFO Queues Deduplication. Reference doc for SQS 5-minute dedup window. docs.aws.amazon.com
• Apache Kafka — Idempotent Producer and Exactly-Once Semantics. The producer-side dedup reference at broker level.
• Brewer E. — CAP Twelve Years Later: How the "Rules" Have Changed, IEEE Computer 2012. Justifies why a multi-tier ledger is a good CAP trade-off for dedup.