Competing Consumers

Several consumers pull from the same channel; the channel hands each message to exactly one of them. This is the horizontal-scaling unit of every EDI hub: to absorb a peak you add workers, not a bigger worker.

Problem

An EDI hub receives hundreds of thousands of messages a day: EDIFACT INVOIC, X12 850, UBL over PEPPOL. A single consumer process cannot hold the throughput, does not survive month-end peaks and becomes a single point of failure. Processing has to scale horizontally: add identical workers that pull in parallel, without rewriting the queue or the producer.

Forces

• Producer throughput can exceed a single consumer's throughput. At 800 messages/sec peak, one parser is under-capacity from the first month-end.
• Bigger boxes cost more than added boxes. Doubling a worker's RAM costs more than doubling the worker count.
• Resilience comes from numbers. If a worker crashes, the remaining N-1 keep draining the queue while the orchestrator restarts one.
• Each message must be processed once. The channel must enforce exclusive delivery; otherwise the burden shifts back to application idempotency.

Solution

EIP §502 (Hohpe & Woolf, 2003) defines Competing Consumers as a point-to-point channel with N consumers, where the broker picks which one receives each message. This is the historical implementation of JMS queues, AMQP work queues (RabbitMQ), Amazon SQS, Azure Service Bus queues, and any Kafka consumer group attached to a multi-partition topic. Each message goes to a single consumer; the consumer's acknowledgement triggers deletion or visibility-lock release.

one queue                              N competing consumers
   ─────────                              ────────────────────

                                       ┌──▶ worker A (parser)
                                       │
   ┌──────────┐    ┌────────────┐      │
   │ producer │ ──▶│   queue    │ ─────┼──▶ worker B (parser)
   └──────────┘    └────────────┘      │
                                       │
                                       └──▶ worker C (parser)

   The queue picks ONE worker per message. Add workers to scale out.

EDI implementation

• Pool of EDIFACT parsers. A queue edi.in.raw receives each interchange as-is on AS2 arrival. A pool of N parsers pulls in parallel, tokenises the interchange into transactions, and pushes to the next queue (edi.in.tx). Auto-scaling provisions more parsers at month-end.
• EN 16931 validation across workers. PEPPOL Schematron BIS 3.0 validation costs hundreds of milliseconds per invoice. A pool of concurrent validators reads peppol.in.validate and writes to a routing queue, turning a p99 cost into additive throughput.
• Outbound AS2 / OFTP2 routing. Every outbound send is I/O-bound (AS2 handshake, encryption, MDN wait). Concurrent workers share a queue edi.out.partner.* and scale linearly as long as the partner keeps up.
• Kafka workers per partition. A topic edi.events.invoic.validated with 12 partitions admits 12 consumers in the same consumer group: this is Competing Consumers at partition granularity, with intra-partition ordering preserved.

Ordering, idempotency and partial FIFO

Message ordering is lost as soon as several consumers compete: two messages pulled in parallel can finish processing in arbitrary order. This is rarely a problem in EDI (each order, each invoice is independent), but it imposes two disciplines:

• Sticky partitioning when per-partner order matters. Kafka and SQS FIFO support partitioning by MessageGroupId (per partner, or per bulk ID) to keep intra-group ordering while parallelising across groups.
• Systematic idempotency at each consumer. When a delivery fails after acknowledgement, some brokers redeliver. See Idempotency.

Anti-patterns

• Single consumer pinned to a queue. Adding only one consumer is just plain point-to-point; no throughput or resilience gain. The pattern only pays off with N ≥ 2 workers.
• Strict ordering across all messages. With strict ordering, you either fall back to a single consumer (losing scale) or partition on a stable attribute. Forcing global order across multiple workers leads to subtle re-ordering bugs.
• Acknowledge before work. ACK-before-work breaks the at-least-once guarantee: if the worker crashes after ACK but before writing, the message is lost silently.
• Sizing on average, not peak. A worker pool tuned for average throughput saturates at month-end and lets the queue balloon to millions of messages before catching up. Auto-scale on queue length, not worker CPU.

Related patterns

• Point-to-Point Channel — the underlying channel, extended to N consumers.
• Message Channel — the generalisation.
• Idempotency — prerequisite as soon as redelivery is possible.
• Dead Letter Channel — destination of messages the workers cannot process.

Sources

• Hohpe G., Woolf B. — Enterprise Integration Patterns, pattern Competing Consumers (§502). enterpriseintegrationpatterns.com — Competing Consumers
• OASIS AMQP 1.0. The RabbitMQ and Azure Service Bus link / credit semantics formalises message allocation to a single subscriber among many. docs.oasis-open.org/amqp
• Apache Kafka — Consumer Groups. Partition allocation to consumers within a group is the modern implementation of the pattern. kafka.apache.org — Consumers
• AWS — SQS FIFO Queues. MessageGroupId enables the intra-group ordering + inter-group scaling trade-off. docs.aws.amazon.com — SQS FIFO