Circuit Breaker

When an EDI partner goes down, we have two bad reflexes: (a) retry in a loop and amplify their outage, (b) give up too early and lose messages. The Circuit Breaker formalises "fail fast, do not hammer, probe cautiously before reconnecting".

Problem

An EDI partner is down — its AS2 endpoint returns 503, its SFTP refuses connections, its broker stops responding. Without precaution, the pipeline will: (a) pile up attempts that all time out, each consuming a thread / a connection; (b) trigger backoff on every send but independently — a thundering herd when the partner comes back; (c) propagate the error to the rest of the platform, which can then drop messages for non-impacted partners. The outage spreads beyond its source.

Forces

• Cost of failing calls. Each call that times out consumes a thread / a connection for 30-60 seconds. With 100 queued messages we saturate the pool in minutes.
• Cascade failure. One partner's outage can block processing of messages bound for other partners sharing the same infrastructure.
• Operational visibility. The "circuit open" state is a clear ops signal for on-call — much more useful than "errors rising".
• Partner asymmetry. Most partners are healthy. Do not apply the same brutal retry to all when only one has trouble.

Solution

Fowler (2014, CircuitBreaker bliki) formalised the pattern previously surfaced by Nygard in Release It! (2007). A state machine per external dependency (per EDI partner), with three states and explicit transitions. In production the circuit breaker lives in mature libraries (resilience4j on the JVM, Polly on .NET, opossum on Node).

┌───────────────────┐
   │      CLOSED      │  ◀────────────┐
   │ (passing calls)  │               │
   └────────┬─────────┘               │
            │ failures >= threshold   │ success in HALF_OPEN
            │ in window               │
            ▼                          │
   ┌───────────────────┐               │
   │      OPEN         │               │
   │ (fail fast)       │               │
   └────────┬──────────┘               │
            │ cooldown elapsed         │
            ▼                          │
   ┌───────────────────┐               │
   │   HALF_OPEN       │ ──────────────┘
   │ (1 probe call)    │
   └────────┬──────────┘
            │ probe fails
            ▼
   back to OPEN

Circuit states

• CLOSED (normal). Calls pass through. Failures are counted over a sliding window (e.g. 10 minutes). If the error rate exceeds a threshold (e.g. 50% over 20 calls), we switch to OPEN.
• OPEN (fail fast). No call is attempted. Every request fails immediately with a CircuitOpenError. The pipeline routes those messages to a patient queue or to the DLC based on criticality.
• HALF_OPEN (probe). After a configured cooldown (1 minute, 5 minutes), one call is allowed through. If that probe call succeeds, we return to CLOSED. If it fails, we go back to OPEN for a new cooldown.

EDI implementation

Two levels of application:

• Per partner. One circuit breaker per (sender, receiver) pair. Lets us isolate one partner's outage without impacting others. The canonical model for an EDI integration hub.
• Per internal dependency. A circuit breaker on the target ERP (SAP, NetSuite, Dynamics). If the ERP is down, we hold integrations in a waiting queue instead of piling up failures.

// Simplified circuit breaker, persistent state per partner
function sendToPartner(partner, message) {
  const cb = getBreaker(partner);

  if (cb.state === 'OPEN') {
    if (now() - cb.openedAt < cb.cooldownMs) {
      throw new CircuitOpenError(partner);
    }
    cb.state = 'HALF_OPEN';
  }

  try {
    const result = transport.send(partner, message);
    if (cb.state === 'HALF_OPEN') {
      cb.state = 'CLOSED';
      cb.failures = 0;
    }
    return result;
  } catch (err) {
    cb.failures++;
    cb.lastFailureAt = now();
    if (cb.failures >= cb.threshold) {
      cb.state = 'OPEN';
      cb.openedAt = now();
      alertOps('circuit opened for ' + partner);
    }
    throw err;
  }
}

In a real EDI context: a partner whose AS2 endpoint returns 503 on 10 consecutive sends triggers the circuit to open. Messages go to a patient queue (partner.X.parked with TTL 24h). After 5 minutes, a probe attempts a send. If it succeeds, we drain the patient queue progressively with rate limiting to avoid saturating the partner who just came back.

Key parameters

• Failure threshold. How many failures to open? Too low (1 failure) triggers false positives; too high (50 failures) waits too long. Typical: 5 to 10 over a 1-minute window, or error rate ≥ 50% over a 20+ call window.
• Cooldown. How long before the probe? Too short (5s) re-stresses a partner that is still down; too long (1h) pointlessly extends downtime when the partner is back. Typical: 1 to 5 minutes.
• Half-open probe count. How many probe calls before closing? 1 is enough in most cases. More, if the partner is known for jagged recoveries.
• Circuit granularity. Per partner, per flow? The (partner, message type) granularity lets us differentiate "ORDERS send fails but INVOIC download works" — useful when the partner has multiple separate endpoints.

Anti-patterns

• Global circuit. A single circuit for all partners: one partner's outage blocks the others. Always split.
• In-memory state only. If circuit state lives in an instance's memory, a restart resets it and the partner is re-hammered. Persist the state to a shared store.
• No visibility. An open circuit that does not alert is a black hole. Always notify ops on open, re-confirm on close.
• Confusing transient and business errors. A 429 (rate limit) or 503 (service unavailable) must count toward the circuit breaker; a 400 (bad request) or 422 (unprocessable entity) are business errors that must not open the circuit — they go to DLC.
• No progressive drain. When the circuit closes, dumping the whole backlog at once re-triggers the outage. Always rate-limit the drain.

Related patterns

• Retry & backoff — covers short transient errors; circuit breaker covers durable outages.
• Dead Letter Channel — where messages go when the circuit stays open past TTL.
• Exception flow — the escalation matrix consuming the circuit-open alerts.

Sources

• Fowler M. — CircuitBreaker, martinfowler.com, 2014. The canonical description of the pattern, picked up by every cloud-native stack since. martinfowler.com/bliki/CircuitBreaker.html
• Nygard M. — Release It! Design and Deploy Production-Ready Software, Pragmatic Bookshelf, 2007 (2nd ed. 2018). The original reference on stability patterns, including the Circuit Breaker.
• Netflix Hystrix (2012, archived). The reference library that popularised the pattern in cloud-native. github.com/Netflix/Hystrix
• resilience4j. Modern successor, reference library on the JVM. resilience4j.readme.io — Circuit Breaker
• AWS Architecture Blog — Timeouts, retries and backoff with jitter (Marc Brooker), with mention of the circuit breaker as a complement. aws.amazon.com — Builders' Library