Transactional Inbox

The consumer records the received message id in an inbox table in the same transaction as the business side effect. If the broker redelivers after a crash, the seen-or-not lookup prevents double execution.

Problem

All modern brokers (Kafka, RabbitMQ, SQS, AS4 eDelivery) guarantee at best at-least-once: after a consumer crash or ack timeout, the message is redelivered. If the consumer processes an InvoiceIssued event by sending an INVOIC to a partner and the process crashes after sending but before broker ack, the retry delivers a second INVOIC — duplicate invoice. The Idempotent Receiver pattern at the business level (UNB control reference) is not enough: you must detect the duplicate before executing the side effect.

Forces

• Broker semantics: at-least-once is universal — exactly-once does not really exist, it is at-least-once + consumer idempotency.
• Critical atomicity: writing the idempotency key and executing the side effect must be atomic, otherwise the pattern breaks.
• Memory footprint: the inbox table grows linearly, must be purged (typically after 30 days, per event partition).
• Cardinality: one inbox per bounded context, indexed by message_id for O(log n) lookup.
• Performance: one extra read per message — acceptable if the table is properly indexed and purged.

Solution

The consumer starts a DB transaction. First step: try to insert the message id into the inbox table (with unique constraint). If insert succeeds, message is new — run the side effect in the same transaction, then commit. If insert fails (unique constraint violation), message is a duplicate — skip the side effect, commit broker ack. The trick: side effect AND "already processed" marker are atomic through the same local DB transaction, without XA.

Structure

Consumer pseudo-code:

while (msg = broker.poll()) {
  BEGIN TX;
  TRY {
    INSERT INTO inbox (message_id, source, received_at)
    VALUES (msg.id, msg.source, now());
    // Success → new message

    // Business side effect in same TX
    INSERT INTO partner_invoices (...) VALUES (...);
    INSERT INTO outbox (...) VALUES (...);

    COMMIT;
    broker.ack(msg);
  } CATCH UNIQUE_VIOLATION {
    // Duplicate → silently skip
    ROLLBACK;  // no side effect
    broker.ack(msg);  // but ack anyway
    metrics.duplicates.inc();
  }
}

EDI implementation

For an EDI hub consumer of Kafka events, the inbox table must uniquely identify each delivered message. On Kafka, the natural key is (topic, partition, offset):

-- PostgreSQL inbox table
CREATE TABLE inbox (
  message_id     VARCHAR(128) PRIMARY KEY,
  source_topic   VARCHAR(80) NOT NULL,
  source_offset  BIGINT NOT NULL,
  consumer_group VARCHAR(80) NOT NULL,
  payload_hash   VARCHAR(64),         -- SHA-256 of payload for drift detection
  received_at    TIMESTAMPTZ DEFAULT now(),
  processed_at   TIMESTAMPTZ,
  partition_key  VARCHAR(80)
);
CREATE INDEX inbox_received_at_idx ON inbox (received_at);

-- Automatic purge after 30 days
DELETE FROM inbox WHERE received_at < now() - INTERVAL '30 days';

-- Transactional Node.js consumer (pseudo)
async function processMessage(msg) {
  const tx = await db.beginTransaction();
  try {
    await tx.query(
      'INSERT INTO inbox (message_id, source_topic, source_offset, consumer_group)
       VALUES ($1, $2, $3, $4)',
      [msg.id, msg.topic, msg.offset, 'edi-invoicer']
    );
    // If we got here, message is new

    await tx.query('INSERT INTO partner_invoices ...');
    await tx.query('INSERT INTO outbox ...');

    await tx.commit();
    await kafkaConsumer.commitOffsets([msg]);
  } catch (err) {
    await tx.rollback();
    if (err.code === '23505') {  // PostgreSQL unique_violation
      // Known duplicate → ack to not see it again
      await kafkaConsumer.commitOffsets([msg]);
      metrics.duplicatesSkipped.inc();
    } else {
      throw err;  // Other error → DLQ
    }
  }
}

For EDI flows ingested from AS4, the AS4 MessageId (UUID in the WS-Addressing header) is the natural key. For EDIFACT, it is the 0020 Interchange Control Reference from UNB combined with (0004 Sender Identification, 0010 Recipient Identification). The inbox must retain history at least as long as the maximum partner retry window (typically 7-30 days).

Anti-patterns

• Checking the inbox after the side effect — loses atomicity, duplicate possible if crash between side effect and inbox insert.
• Inbox in Redis (volatile memory) — a Redis crash loses idempotency, duplicates possible.
• No purge — table grows linearly, performance degrades after a few months.
• Confusing inbox and audit journal — inbox is tactical (anti-duplicate), not legal (see Event Sourcing).
• Not hashing the payload — an attacker could re-submit the same message_id with different payload; the hash detects the drift.

Related patterns

• Inbox (architectural view) — the generic version.
• Transactional Outbox — complementary sender-side pattern.
• Idempotent Receiver EDI — business-level EDI idempotency, complementary to technical inbox.
• Message Deduplication Ledger — cross-broker generalisation of the inbox.
• At-Least-Once Delivery — the broker semantics that make the inbox necessary.

Sources

• Vasters C. — Idempotent Receiver Pattern, Microsoft Azure Architecture. learn.microsoft.com
• Helland P. — Idempotence Is Not a Medical Condition, ACM Queue 2012. The philosophical reference for the pattern. queue.acm.org
• Kleppmann M. — Designing Data-Intensive Applications, O'Reilly 2017, §11.5 ("Faults and Idempotence").
• Confluent — Exactly-Once Semantics Are Possible: Here's How Kafka Does It, 2017. Details why Kafka exactly-once is at-least-once + consumer idempotency.
• RFC 7240 — HTTP Idempotency Key Header Field (Snell, 2014). HTTP norm transposable to message brokers.