11# datamesh-reference-arch-python
22
3- A working data-mesh reference implementation in Python — services across
4- REST, gRPC, GraphQL, and Kafka, deployed via helm, with contracts, a
5- catalog, progressive delivery, autoscaling, and full observability.
3+ A working, runnable reference implementation of a ** data mesh** in Python —
4+ five domain services and a GraphQL gateway across REST, gRPC, GraphQL, and
5+ Kafka, deployed via helm to Kubernetes, with versioned contracts, a data
6+ catalog with cross-product lineage, progressive delivery through a service
7+ mesh, event-driven autoscaling, and end-to-end observability. It fits on a
8+ laptop while remaining recognisable as the same shape you'd run in
9+ production.
610
7- This is the standalone home of the data-mesh reference that previously
8- lived as §17 of [ ` patterncatalyst/minikube-on-fedora ` ] ( https://github.com/patterncatalyst/minikube-on-fedora ) .
9- It contains the complete reading set, the runnable example tree, the
10- presentations, the diagrams, and the historical decision log.
11+ The published site at
12+ [ patterncatalyst.github.io/datamesh-reference-arch-python] ( https://patterncatalyst.github.io/datamesh-reference-arch-python/ )
13+ is the reading set; the tree under ` examples/lgtm-datamesh/ ` is the
14+ runnable system it describes.
15+
16+ ## What this is
17+
18+ Most data-mesh material is conceptual; this one is runnable. And most
19+ runnable material is either too small to show the cross-product behavior
20+ that defines a mesh, or too production-scaled to run on a single host.
21+ This reference threads that gap: it takes Zhamak Dehghani's four
22+ data-mesh principles and shows what each looks like as running pods,
23+ YAML, and live traffic —
24+
25+ - ** Domain ownership** — five domain services (order, inventory, payment,
26+ shipping, notification), each owning its own schema in an
27+ operator-managed Postgres, its own contract, and its own lifecycle.
28+ - ** Data as a product** — each service publishes a versioned, discoverable
29+ interface: OpenAPI for REST, Protobuf for gRPC, GraphQL SDL at the read
30+ surface, and Avro for events — all registered in a schema registry, all
31+ catalogued with ownership and lineage.
32+ - ** Self-serve platform** — the platform capabilities (mesh, autoscaling,
33+ observability, database and Kafka operators) are installed once and
34+ consumed by every product; adding a new data product to the mesh is a
35+ replayable, scripted workflow (` demo-add-data-product.sh ` ).
36+ - ** Federated computational governance** — the contracts registry and the
37+ catalog make the governance surfaces * computational* : schema
38+ compatibility is checked by the registry, and cross-product lineage is
39+ declared and queryable rather than tribal knowledge.
40+
41+ The reading set builds this up over ten pages — concepts, the Kubernetes
42+ substrate, services and data products, contracts and the catalog, the data
43+ planes, progressive delivery with mTLS, elasticity, observability,
44+ anti-patterns, and a principle-by-principle summary.
45+
46+ ## What it demonstrates
47+
48+ The system's headline behaviors, each runnable as a demo (they are the
49+ five acts of ` demos/walkthrough.sh ` , plus the two phase demos):
50+
51+ 1 . ** A trace across products** — one GraphQL query at the gateway fans out
52+ to order-service (REST) and inventory-service (gRPC), and all three
53+ spans land in Tempo stitched by a shared trace id. The mesh's
54+ interesting behavior lives * between* products; this makes it legible.
55+ 2 . ** Elastic data products** — KEDA scales notification-service on Kafka
56+ consumer lag (zero → up → zero: an event consumer runs only as much as
57+ its backlog warrants) and the GraphQL gateway on HTTP demand (scale to
58+ zero when idle, wake on the first request).
59+ 3 . ** Canary of a contract, not just a binary** — order-service evolves its
60+ API v1 → v2 (an additive ` currency ` field) and Istio shifts a
61+ controllable fraction of live traffic to the new contract version —
62+ 90/10, then 50/50, then a clean rollback — with no flag-day break.
63+ 4 . ** Cross-product lineage** — OpenMetadata shows the operational spine
64+ (` orders ` table → ` order-placed ` topic → ` notifications ` table) as
65+ declared, queryable lineage across three products' boundaries.
66+ 5 . ** Live mesh topology** — Kiali renders the running mesh graph with the
67+ canary split visible, backed by the same Prometheus the dashboards use.
68+
69+ Plus the two replayable workflow demos: ** adding a data product to the
70+ mesh** (deploy review-service → publish its contract → catalog it with
71+ lineage → back it all out), and the ** contract canary** as a
72+ presenter-driven up/shift/down cycle.
73+
74+ Every claim is smoke-tested: 19 ` smoke-*.sh ` scripts cover each capability
75+ end to end, verified order-independent (see
76+ ` _plans/reconciliation.md ` for what's been cluster-verified and when).
77+
78+ ## Tech stack
79+
80+ | Layer | Technology | Role |
81+ | ---| ---| ---|
82+ | Services | Python 3.12, FastAPI, SQLAlchemy + Alembic, Poetry | Five domain services + GraphQL gateway; migrations via init-containers |
83+ | Protocols | REST (OpenAPI), gRPC (Protobuf/buf), GraphQL (SDL), Kafka (Avro) | The deliberate protocol mix: cross-product APIs, hot paths, read surface, async spine |
84+ | Packaging | Helm (umbrella chart + per-service subcharts), podman | One chart tree for the whole system; images built on the host, pushed to the in-cluster registry |
85+ | Substrate | Kubernetes v1.32 on minikube (rootless podman driver, containerd) | The whole system on one 24 GB profile |
86+ | Database | CloudNativePG operator, PostgreSQL | One shared cluster, schema-per-service ownership |
87+ | Events | Strimzi operator, Apache Kafka | The asynchronous spine (` order-placed ` topic) |
88+ | Contracts | Apicurio Registry | OpenAPI + Protobuf + GraphQL SDL (discovery) and Avro (runtime, Confluent-compatible API) |
89+ | Catalog | OpenMetadata (+ OpenSearch) | Service inventory, ownership, cross-product lineage |
90+ | Mesh | Istio | mTLS, traffic splitting for the v1→v2 contract canary |
91+ | Autoscaling | KEDA (core + HTTP add-on) | Consumer-lag scaling for event consumers, request-based scale-to-zero for the gateway |
92+ | Observability | Prometheus, Tempo, Grafana, Kiali, OpenTelemetry | Metrics, cross-product traces, dashboards, live mesh topology |
93+ | Site | Jekyll / GitHub Pages | The ten-page reading set with paired SVG + Excalidraw diagrams |
1194
1295## What's here
1396
@@ -24,16 +107,31 @@ presentations, the diagrams, and the historical decision log.
24107 Historical material from the capstone era is preserved under
25108 ` _plans/archive/ ` .
26109- ** ` scripts/ ` ** — top-level helper scripts: cross-reference and Liquid
27- collision linters, setup helpers for Istio / KEDA / Strimzi, and
28- the diagram-splicing utility.
110+ collision linters, a Fedora prerequisites audit, setup helpers for
111+ Istio / KEDA / Strimzi, and the diagram-splicing utility.
112+ - ** ` onboarding/ ` ** — the orientation guide and the lessons-learned
113+ distillation.
29114
30115## Getting started
31116
32- Read [ ` onboarding/GETTING-STARTED.md ` ] ( onboarding/GETTING-STARTED.md ) for
33- the orientation guide. The published site at
34- [ patterncatalyst.github.io/datamesh-reference-arch-python] ( https://patterncatalyst.github.io/datamesh-reference-arch-python/ )
35- is the canonical reading order; the runnable example tree under
36- ` examples/lgtm-datamesh/ ` is the canonical build target.
117+ ** To read:** start at the published site's
118+ [ reading guide] ( https://patterncatalyst.github.io/datamesh-reference-arch-python/docs/00-index/ ) ,
119+ or read [ ` onboarding/GETTING-STARTED.md ` ] ( onboarding/GETTING-STARTED.md )
120+ for repo orientation.
121+
122+ ** To run:** the operational entry point is
123+ [ ` examples/lgtm-datamesh/README.md ` ] ( examples/lgtm-datamesh/README.md ) —
124+ check the prerequisites there (or run ` ./scripts/audit-fedora-prereqs.sh `
125+ for a read-only host check), then:
126+
127+ ``` bash
128+ cd examples/lgtm-datamesh
129+ ./scripts/bootstrap-capstone.sh # the whole stack, ten gated tiers
130+ ./demos/walkthrough.sh # the five-act presenter walkthrough
131+ ```
132+
133+ The verified configuration is Fedora with rootless podman, 64 GB RAM
134+ (24 GB for the minikube profile), and minikube ≥ 1.36.
37135
38136## Origin
39137
0 commit comments