Retry, exponential backoff and circuit breaker

When a partner doesn't respond, retrying is tempting and dangerous. Five rules borrowed from TCP, AWS and Martin Fowler are enough to keep a partner incident from turning into a platform-wide outage.

Problem

An AS2 send times out. AS4 returns a 500. OFTP2 closes the session on abort. Now what? The naive answer is to retry immediately. That is exactly how retry storms are born: every pending send, the moment it is released, simultaneously hammers an already-saturated partner, deepening the outage for everyone — including the sender, whose queues then explode.

Forces

• Partner outages are usually temporary. Common causes: maintenance, expired certificate, traffic spike. The outage is rarely permanent.
• Amplification is real. 1,000 in-flight sends retrying every 30 seconds means 33 req/s on a partner that is already struggling. That alone can prevent recovery.
• Waiting also costs. On the sender side, holding sends in flight for hours consumes storage and resources. A ceiling is required.
• Some messages must not be lost. A B2G invoice requires a trace. Beyond N retries, the message must land in a DLQ with an alert — never silently dropped.

Exponential backoff and jitter

The canonical pattern goes back to RFC 5681 (TCP Congestion Control): on each failure, wait 2n · base before the next attempt. For typical EDI sends: base = 1 s, ceiling = 30 min, 7 attempts — that is 1, 2, 4, 8, 16, 32, 64 seconds (then capped). Two minutes of retries in the early phase, then spaced.

Backoff alone is not enough. If all queued sends were blocked on the same outage (incident at T0), they all retry at T0+1, T0+3, T0+7 — in phase. The spike recurs at every step. Adding random jitter breaks that synchronisation. Marc Brooker's reference article (AWS, 2015) shows that full jitter (multiply the interval by a random number between 0 and 1) typically beats equal jitter or no jitter at all, because it flattens the attempt distribution entirely.

Circuit breaker

The circuit breaker (Martin Fowler, 2014) cuts off transmission entirely when a partner is clearly down. It has three states:

• Closed — normal flow. Failures are counted.
• Open — past a threshold (e.g. 5 failures in 60 seconds), flip to Open. Every subsequent send to this partner lands in the DLQ with a circuit-open note. The partner is not contacted during the cooldown (5 to 30 minutes typically).
• Half-Open — after cooldown, let one probe through. If it succeeds → Closed. If it fails → Open for another cooldown.

Circuit breakers protect both sides: they stop the platform from queueing useless sends, and they stop pummelling a partner while it tries to recover. That is why they are more useful than people expect in EDI, where partners are often fragile legacy systems.

Dead-letter queue

The dead-letter queue (Hohpe & Woolf, 2003) is where messages end up after exhausting retries or after a non-retryable failure (4xx error, invalid signature, corrupt payload). It is a persistent visible queue — not just an error log. Three essential properties:

• Persistence. A platform restart loses nothing. The DLQ is typically on disk or in a backed-up database.
• Replay-friendly. An operator must be able to re-inject a DLQ message in one click once the partner is back up.
• Error metadata. Store last error, timestamp, attempt count, partner id — to make ops triage tractable.

Applied to EDI flows

Policy in pseudo-code

// Retry policy — pseudo-code
const MAX_ATTEMPTS = 7;
const BASE_DELAY_MS = 1_000;          // 1 s
const MAX_DELAY_MS = 30 * 60 * 1_000; // 30 min ceiling
const JITTER_RATIO = 0.5;             // ±50 %

async function sendWithRetry(message) {
  for (let attempt = 0; attempt < MAX_ATTEMPTS; attempt++) {
    if (circuitBreakerOpen(partner)) {
      await deadLetterQueue.push(message, 'circuit-open');
      return;
    }

    try {
      const ack = await sendOverAs2(message);
      recordSuccess(partner);
      return ack;
    } catch (err) {
      recordFailure(partner, err);
      if (!isRetryable(err)) {
        await deadLetterQueue.push(message, err.code);
        return;
      }
    }

    // exponential backoff with full jitter
    const exp = Math.min(BASE_DELAY_MS * 2 ** attempt, MAX_DELAY_MS);
    const jitter = exp * (Math.random() * JITTER_RATIO * 2 - JITTER_RATIO);
    await sleep(exp + jitter);
  }

  await deadLetterQueue.push(message, 'max-attempts');
}

Anti-patterns

• Immediate retry without backoff. Pounds the partner problem at full power. Never do this.
• Uniform retry across error types. An invalid signature won't go away by retrying. Distinguish retryable (timeout, 5xx) from non-retryable (4xx, 401/403, signature/format).
• Infinite retry. No ceiling = exploding queues and escalating incidents. Always cap N and route to DLQ.
• Over-sensitive circuit breaker. With a threshold of 2 failures, every micro-incident flips to Open and blocks everything. Keep the threshold above normal partner variability.
• Silent DLQ. A DLQ message that doesn't alert anyone sits there until a customer calls. The DLQ must always emit an outbound alert (mail, Slack, ticket).
• Backoff without jitter. Every send syncs on the same steps and reproduces the spike. Always add jitter.

Related patterns

• Idempotency — indispensable prerequisite to retry.
• Acknowledgements — to know when to retry (on which missing ACK level).
• Exception flow — the escalation matrix when retries are exhausted.

Sources

• RFC 5681 — TCP Congestion Control (Allman, Paxson, Blanton, September 2009). The canonical reference on exponential backoff. rfc-editor.org/rfc/rfc5681
• AWS Architecture Blog — "Exponential Backoff And Jitter" (Marc Brooker, 2015). Quantitative comparison of no-jitter / equal-jitter / full-jitter on 100,000 simulated retries. aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter
• Fowler M. — Circuit Breaker (martinfowler.com, 2014). The canonical description. martinfowler.com/bliki/CircuitBreaker.html
• Hohpe G., Woolf B. — Enterprise Integration Patterns, Dead Letter Channel and Guaranteed Delivery. enterpriseintegrationpatterns.com — Dead Letter Channel
• RFC 6749 §5.2 — OAuth 2.0 error responses, which codifies the retryable / non-retryable distinction at the HTTP layer.