Enrichment Pipeline

The ordered enrichment chain turning a raw parsed message into a payload usable by the business — each step consults a reference, adds a field.

Problem

An EDIFACT INVOIC contains codes: supplier GLN (3010234567890), product GTIN (0023456789012), bank BIC (BNPAFRPP). For the ERP, those codes are opaque: it needs the supplier name, billing address, product label, applicable VAT, IBAN to credit, country code, customs HS classification. If each downstream consumer (ERP, archive, lake) performs its own lookups, we duplicate 5 times the requests to reference data, lose consistency, and couple every consumer to reference schemas.

Forces

• EDI messages are intentionally coded. Historical size optimisation, identifier normalisation.
• Consumers need the decoded form. End-users do not read 3010234567890, they want "Carrefour France SAS".
• References are the source of truth. PIM, MDM, legal registry — single centralised lookup.
• Steps have dependencies. VAT calculation depends on supplier country resolved in step 1.
• References can be slow or unavailable. The pipeline must handle degraded mode.

Solution

Compose a pipeline of Content Enricher stages in ordered stages. Each stage takes the message + previous enrichments as input, performs its lookup, adds its field, passes to next. The order follows dependencies. Each stage is independently testable. Resilience is per stage: if PIM is down, continue with productLabel: null and flag the message. The pipeline is versioned and observable per step (latency, failure rate, cache hit). Implementations: Apache Camel routes, Kafka Streams topology, AWS Step Functions, Temporal workflows.

EDIFACT INVOIC (raw, GLN codes only)
        │
        ▼
  ┌──────────────────────────┐
  │ 1. Enrich Partner GLN    │ ─── lookup GLN → Partner master
  │    + supplier name/addr  │
  └─────────┬────────────────┘
            ▼
  ┌──────────────────────────┐
  │ 2. Enrich Product GTIN   │ ─── lookup PIM
  │    + label, weight, HS   │
  └─────────┬────────────────┘
            ▼
  ┌──────────────────────────┐
  │ 3. Enrich Tax rates      │ ─── lookup country / product
  │    + VAT rate, code      │
  └─────────┬────────────────┘
            ▼
  ┌──────────────────────────┐
  │ 4. Enrich BIC bank info  │ ─── lookup BIC → IBAN validation
  └─────────┬────────────────┘
            ▼
  Enriched INVOIC → broker → ERP

EDI implementation

Concrete case: hospital EDI hub receives an ORM-O01 HL7 v2.5 message with MRN and LOINC codes. The enrichment pipeline: (1) MRN → patient identity from MPI (Master Patient Index); (2) LOINC code → exam label from FHIR Terminology Server; (3) ICD-10 diagnosis code → clinical label; (4) prescriber NPI → contacts and specialty; (5) destination pharmacy → address, supported formats. The resulting message is usable by the HIS without new lookups. Retail: EDIFACT ORDERS with GTIN only, enriched by PIM (GS1 GDSN) to retrieve label, weight, packaging, HS code, customs rules. Camel route: from("kafka:edi.in").to("bean:partnerEnricher").to("bean:productEnricher").to("bean:taxEnricher").to("kafka:edi.enriched"); Each step uses local cache (Caffeine) + cluster cache (Redis) to avoid repeat queries against the master reference.

Anti-patterns

• Downstream enrichment at each consumer. Lookup duplication, result divergence if reference changes between two consumers.
• Synchronous blocking stage without timeout. If PIM stalls 30s, the whole pipeline blocks.
• No cache. The same GLN looked up 1,000 times per second: useless requests, high latency.
• No gap handling. If supplier GLN is not in the reference, we crash instead of flagging and routing to Dead Letter.
• Enricher containing business logic. "If country = FR, tax 20%" in the enricher: business logic to place in a dedicated service.
• Giant pipeline. 20 stages: latency and complexity. Group logically related stages.

Related patterns

• Content Enricher — the unit brick of the pipeline.
• Pipes and Filters — the general topology (stage chain).
• Validation Pipeline — sibling pattern, to run before or after enrichment.
• Claim Check — for heavy enrichments (product photo), store and reference.
• Composed Message Processor — structured variant for a composite message enriched per line.

Sources

• Hohpe G., Woolf B. — Enterprise Integration Patterns, Addison-Wesley 2003. Content Enricher (EIP §336). enterpriseintegrationpatterns.com — DataEnricher
• GS1 — GDSN Global Data Synchronization Network. The global product reference for GTIN enrichment. gs1.org — GDSN
• FHIR Terminology Service. The lookup service for LOINC, SNOMED, ICD-10 in healthcare. hl7.org — Terminology Service
• Apache Camel — Enricher EIP. Java reference implementation. camel.apache.org — Content Enricher
• Confluent — Schema Registry + Stream enrichment. The classic Kafka Streams enricher pattern. confluent.io — Streams concepts