Chaos Engineering

Deliberately injecting failures into an EDI system to verify it resists. The discipline that turns resilience from a hope into a measured property — practice born at Netflix with Chaos Monkey.

Problem

A modern EDI hub integrates dozens of resilience patterns: circuit breaker, retry, bulkhead, dead letter channel, idempotency. But the resilience of a distributed system is not the sum of its components' resiliences — it emerges from interactions, and these emergences are often surprising in real failure. "Paper" resilience is different from "production" resilience — the famous sound and fury of any distributed production. The only way to be confident is to test.

Forces

• Failures are inevitable: network, partner, AS2 down, expired certificate, saturated queue — everything happens sooner or later.
• Testing in non-prod is not enough: load, topology, dependencies differ in prod.
• The cost of discovery via real incident is high: MTTR, SLA penalties, reputation damage.
• Experimentation must be controlled: limited scope, automatic rollback, blast-radius plan.
• Learning is the main objective: each experiment produces post-mortems and improves the system.

Solution

Formalise chaos experiments: verifiable hypothesis, delimited population, injected perturbation, steady-state measurement, conclusion. The cycle follows the Principles of Chaos Engineering published by the Netflix team: (1) define a measurable steady state, (2) hypothesise that this state persists under perturbation, (3) vary real events (kill instance, latency injection, network partition), (4) confirm or refute the hypothesis in controlled production. Tools: Chaos Monkey, Chaos Kong, Gremlin, AWS Fault Injection Simulator, Chaos Mesh (Kubernetes), Litmus (Kubernetes).

Chaos experimentation cycle:

   1. Hypothesis: "If the AS2 partner does not answer for 60s,
                   retries respect exponential backoff and no
                   message is lost."
              │
              ▼
   2. Population: 1 pilot partner (non-critical)
              │
              ▼
   3. Variable: kill TCP between hub and partner for 60s
              │
              ▼
   4. Measurement (steady state):
      • Latency p95 (before, during, after)
      • Lost message rate
      • Circuit breaker behaviour
      • Retries executed
              │
              ▼
   5. Validation:
      • If hypothesis verified → expand scope
      • Else → bug detected, fix + new test

EDI implementation

Concrete case: validate resilience of the inbound Walmart AS2 flow. Hypothesis: "if the internal validation pipeline goes down, messages are queued for retry, then routed once the pipeline recovers, with no loss or duplication". Controlled experimentation: every Friday 14:00-15:00 in pre-prod, the validator is killed via Chaos Mesh for 5 min. Measurement: rate of received / acknowledged / routed-to-ERP messages. Expected result: 100% received, 0% lost, routing resumed after Kubernetes replica. In real production: weekly programmes on progressively expanded scope: simulated expired certificate, AS2 partner timing out, killed Kafka broker, retry storms. Each experiment documents a game day, a post-mortem, system amendments. The modern Continuous Verification approach integrates these tests into CI/CD. Common EDI tools: Chaos Mesh on Kubernetes, Toxiproxy for latency injection on AS2 sockets.

Anti-patterns

• Chaos without hypothesis: "we break to see" = controlled vandalism, not experimentation.
• No limited blast radius: first experiment at full scale = guaranteed incident.
• No steady-state measurement: without baseline, impossible to say whether the experiment validated or not.
• No automatic rollback: if the experiment goes off, an instant kill switch is needed.
• Chaos in prod for a system without baseline resilience: start with the fundamentals (circuit breaker, retry, idempotency) before injecting chaos, otherwise everything breaks.
• Isolated game days without follow-up: if learnings are not tracked and corrected, the benefit evaporates.

Related patterns

• Circuit Breaker — pattern classically tested by chaos.
• Bulkhead — pattern whose isolation is validated by chaos.
• Retry with backoff — pattern whose behaviour is validated by chaos.
• Health Check — pattern whose reliability is verified by chaos.
• Canary Release — step that follows chaos validation of a new version.

Sources

• Rosenthal C., Hochstein L., Blohowiak A., Jones N., Basiri A. — Chaos Engineering: System Resiliency in Practice, O'Reilly 2017 (2nd ed. 2020). Reference work by the founding Netflix team.
• Principles of Chaos Engineering: the five principles formalised by the Netflix team that define the discipline. principlesofchaos.org
• Netflix Tech Blog — Chaos Monkey. The original publication (2010) of the tool that initiated the movement. netflix.github.io — chaosmonkey
• Chaos Mesh (CNCF Sandbox) — Documentation of the open-source chaos engineering platform for Kubernetes. chaos-mesh.org
• AWS Fault Injection Simulator (FIS). AWS managed service to orchestrate chaos experiments. aws.amazon.com — fis
• Nygard M. — Release It! Design and Deploy Production-Ready Software, Pragmatic Bookshelf, 2nd ed. 2018. Discussion of stress testing and resilience.