Skip to content

Commit 7d1912a

Browse files
docs: enrich the top-level README — what this is, stack, capabilities
Grow the README from a directory listing into a real front door, drawn from the PRD's framing and the reading set's structure: - What this is: the runnable-vs-conceptual gap the reference threads, and each of Dehghani's four principles mapped to its concrete implementation in this system - What it demonstrates: the five walkthrough acts (trace across products, elastic data products, canary of a contract, cross-product lineage, live mesh topology) plus the two replayable workflow demos, with a pointer to the reconciliation file for verification state - Tech stack: a layer-by-layer table from the Python services through protocols, operators, mesh, autoscaling, and observability to the Jekyll site - Getting started: split into read (site) and run (bootstrap + walkthrough, with the prerequisites audit) paths Existing What's-here and Origin sections kept, lightly refreshed (scripts/ bullet now mentions the prerequisites audit; onboarding/ bullet added).
1 parent 750e9d5 commit 7d1912a

1 file changed

Lines changed: 112 additions & 14 deletions

File tree

‎README.md‎

Lines changed: 112 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,96 @@
11
# datamesh-reference-arch-python
22

3-
A working data-mesh reference implementation in Python — services across
4-
REST, gRPC, GraphQL, and Kafka, deployed via helm, with contracts, a
5-
catalog, progressive delivery, autoscaling, and full observability.
3+
A working, runnable reference implementation of a **data mesh** in Python —
4+
five domain services and a GraphQL gateway across REST, gRPC, GraphQL, and
5+
Kafka, deployed via helm to Kubernetes, with versioned contracts, a data
6+
catalog with cross-product lineage, progressive delivery through a service
7+
mesh, event-driven autoscaling, and end-to-end observability. It fits on a
8+
laptop while remaining recognisable as the same shape you'd run in
9+
production.
610

7-
This is the standalone home of the data-mesh reference that previously
8-
lived as §17 of [`patterncatalyst/minikube-on-fedora`](https://github.com/patterncatalyst/minikube-on-fedora).
9-
It contains the complete reading set, the runnable example tree, the
10-
presentations, the diagrams, and the historical decision log.
11+
The published site at
12+
[patterncatalyst.github.io/datamesh-reference-arch-python](https://patterncatalyst.github.io/datamesh-reference-arch-python/)
13+
is the reading set; the tree under `examples/lgtm-datamesh/` is the
14+
runnable system it describes.
15+
16+
## What this is
17+
18+
Most data-mesh material is conceptual; this one is runnable. And most
19+
runnable material is either too small to show the cross-product behavior
20+
that defines a mesh, or too production-scaled to run on a single host.
21+
This reference threads that gap: it takes Zhamak Dehghani's four
22+
data-mesh principles and shows what each looks like as running pods,
23+
YAML, and live traffic —
24+
25+
- **Domain ownership** — five domain services (order, inventory, payment,
26+
shipping, notification), each owning its own schema in an
27+
operator-managed Postgres, its own contract, and its own lifecycle.
28+
- **Data as a product** — each service publishes a versioned, discoverable
29+
interface: OpenAPI for REST, Protobuf for gRPC, GraphQL SDL at the read
30+
surface, and Avro for events — all registered in a schema registry, all
31+
catalogued with ownership and lineage.
32+
- **Self-serve platform** — the platform capabilities (mesh, autoscaling,
33+
observability, database and Kafka operators) are installed once and
34+
consumed by every product; adding a new data product to the mesh is a
35+
replayable, scripted workflow (`demo-add-data-product.sh`).
36+
- **Federated computational governance** — the contracts registry and the
37+
catalog make the governance surfaces *computational*: schema
38+
compatibility is checked by the registry, and cross-product lineage is
39+
declared and queryable rather than tribal knowledge.
40+
41+
The reading set builds this up over ten pages — concepts, the Kubernetes
42+
substrate, services and data products, contracts and the catalog, the data
43+
planes, progressive delivery with mTLS, elasticity, observability,
44+
anti-patterns, and a principle-by-principle summary.
45+
46+
## What it demonstrates
47+
48+
The system's headline behaviors, each runnable as a demo (they are the
49+
five acts of `demos/walkthrough.sh`, plus the two phase demos):
50+
51+
1. **A trace across products** — one GraphQL query at the gateway fans out
52+
to order-service (REST) and inventory-service (gRPC), and all three
53+
spans land in Tempo stitched by a shared trace id. The mesh's
54+
interesting behavior lives *between* products; this makes it legible.
55+
2. **Elastic data products** — KEDA scales notification-service on Kafka
56+
consumer lag (zero → up → zero: an event consumer runs only as much as
57+
its backlog warrants) and the GraphQL gateway on HTTP demand (scale to
58+
zero when idle, wake on the first request).
59+
3. **Canary of a contract, not just a binary** — order-service evolves its
60+
API v1 → v2 (an additive `currency` field) and Istio shifts a
61+
controllable fraction of live traffic to the new contract version —
62+
90/10, then 50/50, then a clean rollback — with no flag-day break.
63+
4. **Cross-product lineage** — OpenMetadata shows the operational spine
64+
(`orders` table → `order-placed` topic → `notifications` table) as
65+
declared, queryable lineage across three products' boundaries.
66+
5. **Live mesh topology** — Kiali renders the running mesh graph with the
67+
canary split visible, backed by the same Prometheus the dashboards use.
68+
69+
Plus the two replayable workflow demos: **adding a data product to the
70+
mesh** (deploy review-service → publish its contract → catalog it with
71+
lineage → back it all out), and the **contract canary** as a
72+
presenter-driven up/shift/down cycle.
73+
74+
Every claim is smoke-tested: 19 `smoke-*.sh` scripts cover each capability
75+
end to end, verified order-independent (see
76+
`_plans/reconciliation.md` for what's been cluster-verified and when).
77+
78+
## Tech stack
79+
80+
| Layer | Technology | Role |
81+
|---|---|---|
82+
| Services | Python 3.12, FastAPI, SQLAlchemy + Alembic, Poetry | Five domain services + GraphQL gateway; migrations via init-containers |
83+
| Protocols | REST (OpenAPI), gRPC (Protobuf/buf), GraphQL (SDL), Kafka (Avro) | The deliberate protocol mix: cross-product APIs, hot paths, read surface, async spine |
84+
| Packaging | Helm (umbrella chart + per-service subcharts), podman | One chart tree for the whole system; images built on the host, pushed to the in-cluster registry |
85+
| Substrate | Kubernetes v1.32 on minikube (rootless podman driver, containerd) | The whole system on one 24 GB profile |
86+
| Database | CloudNativePG operator, PostgreSQL | One shared cluster, schema-per-service ownership |
87+
| Events | Strimzi operator, Apache Kafka | The asynchronous spine (`order-placed` topic) |
88+
| Contracts | Apicurio Registry | OpenAPI + Protobuf + GraphQL SDL (discovery) and Avro (runtime, Confluent-compatible API) |
89+
| Catalog | OpenMetadata (+ OpenSearch) | Service inventory, ownership, cross-product lineage |
90+
| Mesh | Istio | mTLS, traffic splitting for the v1→v2 contract canary |
91+
| Autoscaling | KEDA (core + HTTP add-on) | Consumer-lag scaling for event consumers, request-based scale-to-zero for the gateway |
92+
| Observability | Prometheus, Tempo, Grafana, Kiali, OpenTelemetry | Metrics, cross-product traces, dashboards, live mesh topology |
93+
| Site | Jekyll / GitHub Pages | The ten-page reading set with paired SVG + Excalidraw diagrams |
1194

1295
## What's here
1396

@@ -24,16 +107,31 @@ presentations, the diagrams, and the historical decision log.
24107
Historical material from the capstone era is preserved under
25108
`_plans/archive/`.
26109
- **`scripts/`** — top-level helper scripts: cross-reference and Liquid
27-
collision linters, setup helpers for Istio / KEDA / Strimzi, and
28-
the diagram-splicing utility.
110+
collision linters, a Fedora prerequisites audit, setup helpers for
111+
Istio / KEDA / Strimzi, and the diagram-splicing utility.
112+
- **`onboarding/`** — the orientation guide and the lessons-learned
113+
distillation.
29114

30115
## Getting started
31116

32-
Read [`onboarding/GETTING-STARTED.md`](onboarding/GETTING-STARTED.md) for
33-
the orientation guide. The published site at
34-
[patterncatalyst.github.io/datamesh-reference-arch-python](https://patterncatalyst.github.io/datamesh-reference-arch-python/)
35-
is the canonical reading order; the runnable example tree under
36-
`examples/lgtm-datamesh/` is the canonical build target.
117+
**To read:** start at the published site's
118+
[reading guide](https://patterncatalyst.github.io/datamesh-reference-arch-python/docs/00-index/),
119+
or read [`onboarding/GETTING-STARTED.md`](onboarding/GETTING-STARTED.md)
120+
for repo orientation.
121+
122+
**To run:** the operational entry point is
123+
[`examples/lgtm-datamesh/README.md`](examples/lgtm-datamesh/README.md) —
124+
check the prerequisites there (or run `./scripts/audit-fedora-prereqs.sh`
125+
for a read-only host check), then:
126+
127+
```bash
128+
cd examples/lgtm-datamesh
129+
./scripts/bootstrap-capstone.sh # the whole stack, ten gated tiers
130+
./demos/walkthrough.sh # the five-act presenter walkthrough
131+
```
132+
133+
The verified configuration is Fedora with rootless podman, 64 GB RAM
134+
(24 GB for the minikube profile), and minikube ≥ 1.36.
37135

38136
## Origin
39137

0 commit comments

Comments
 (0)