ediverse

Type to search across standards, messages, tools and glossary.

ediverse Explore the platform

Spotlight PEPPOL BIS Billing 3.0 The EU e-invoicing mandate is here — France Sept 2026, Belgium Jan 2026, Germany 2025.

— 01

Foundations

— 02

Standards

— 03

Flagship messages

— 04

Tools

Sharding

Split data and traffic across N independent instances instead of fitting everything onto one.

Problem

A service or DB scales linearly with resources up to the limits of a single instance (CPU, RAM, IOPS, bandwidth). Vertical scale costs exponentially more beyond certain thresholds and caps out. Horizontal scale via sharding distributes load but introduces new constraints: cross-shard queries, distributed transactions, rebalancing.

Forces

  • A single instance has a reachable CPU/RAM/IOPS ceiling.
  • A too-large partition (hot shard) cancels the sharding benefit.
  • Cross-shard queries are costly or even impossible.
  • Distributed transactions (2PC, Saga) must replace local transactions.
  • Rebalancing moves data and impacts performance.

Solution

Partition data by a shard key that determines the target shard. Each shard is a complete, autonomous instance hosting a strictly disjoint data subset. A router (proxy or client library) computes the shard from the key. Three dominant strategies exist (below). The pattern fits DBs (Postgres Citus, MongoDB, Cassandra), caches (Redis Cluster), queues (Kafka partitions), and application services (one EDI hub per region).

Sharding strategies

  • Hash-based — shard = hash(key) % N or via consistent hashing. Uniform distribution but range queries impossible. Good for user_id, partner_id.
  • Range-based — each shard hosts a key interval (by date, by region). Allows range queries but risks hot shards (the latest time partition takes all writes).
  • Lookup-based — a routing table (shard_map) maps key → shard. Very flexible (single-key move possible), but lookup adds indirection and the table must be HA.

EDI implementation

A multi-tenant high-volume EDI hub shards per tenant: shard = hash(partnerId). Each shard owns its Postgres DB and Kafka topic set. Consequence: a chaotic partner (Walmart on Black Friday peak hour) does not bring down others. The rule: always shard on a key that contains the tenant — orders, invoices, AS2 messages. For global queries (hub-wide KPIs), aggregate asynchronously into a data lake (cf. Data Mesh).

Anti-patterns

  • Sharding without hot key analysis — a premium account brings down its shard.
  • Shard key that changes during a row's lifetime — object must be moved.
  • Frequent cross-shard queries — cancels partition gain (costly fan-out).
  • Blocking synchronous rebalancing — an online mechanism is required.
  • Too many shards (over-sharding) — coordination overhead exceeds the gain.
  • Consistent Hashing — typical algorithm for hash-based sharding.
  • Bulkhead — logical isolation between shards.
  • Saga Orchestration — replaces synchronous distributed transactions.
  • CQRS — often combined; write shard differs from read shard.

Sources