Skip to content

Commit c7bab7f

Browse files
r0.1: editorial pass on PRD, GETTING-STARTED, LESSONS-LEARNED + adopt summary section
- PRD.md full edit: three-audience analysis, explicit Kubernetes-as-model / minikube-as-verified-runtime / OpenShift-deck-pairs-it positioning, non-goals honest about exclusions - onboarding/GETTING-STARTED.md full edit: 'What this is and isn't' lede, three-audience entry points, contribution path for non-minikube runtimes - onboarding/LESSONS-LEARNED.md full edit: 5 architectural lessons, 8 mechanical gotchas, 5 process lessons, plus 'what I would do differently' - New _docs/10-summary.md: closing summary by principle, adopts all 5 deck-only diagrams (17-value-*.svg, 17-trusted-supply-chain.svg) with prose contextualization of their OpenShift vocabulary - 00-index, index.html, README.md adjusted for new page count (ten pages, four movements) - Cross-reference linter clean (68 links); Liquid linter clean
1 parent d9c29f1 commit c7bab7f

7 files changed

Lines changed: 1046 additions & 196 deletions

File tree

‎PRD.md‎

Lines changed: 346 additions & 95 deletions
Large diffs are not rendered by default.

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ presentations, the diagrams, and the historical decision log.
1111

1212
## What's here
1313

14-
- **`_docs/`** — the nine reading-set pages (00-index through 09-anti-patterns).
14+
- **`_docs/`** — the ten reading-set pages (00-index through 10-summary).
1515
The published site renders these at `https://patterncatalyst.github.io/datamesh-reference-arch-python/docs/<name>/`.
1616
- **`examples/lgtm-datamesh/`** — the runnable code: five domain services
1717
plus a GraphQL gateway, helm charts, Kafka and Postgres operators,

‎_docs/00-index.md‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -18,14 +18,16 @@ page.
1818

1919
## How the set is organized
2020

21-
The nine pages fall into three movements. The **conceptual grounding** (pages 1–2)
21+
The ten pages fall into four movements. The **conceptual grounding** (pages 1–2)
2222
establishes what a data mesh is and why Kubernetes is a natural substrate for one. The
2323
**implementation** (pages 3–8) builds the system: the services and their data products,
2424
the contracts and catalog that make them discoverable, the data planes they communicate
2525
over, progressive delivery with mutual TLS, elastic scaling and recovery, and the
26-
observability to see it all. The closing page (9) steps back to the **failure modes** —
27-
the conceptual and organizational anti-patterns that derail data-mesh efforts even when
28-
the technology is sound.
26+
observability to see it all. The **failure modes** (page 9) steps back to the
27+
conceptual and organizational anti-patterns that derail data-mesh efforts even when
28+
the technology is sound. The **summary** (page 10) closes by reorganizing the same
29+
material by principle — for each of the four principles, the value it delivers and the
30+
implementation pieces that realize it.
2931

3032
## The pages
3133

@@ -56,6 +58,9 @@ the technology is sound.
5658
- [**9 · Anti-patterns**]({{ '/docs/09-anti-patterns/' | relative_url }}) —
5759
The conceptual and organizational ways data-mesh efforts go wrong, drawn from the
5860
literature, so you can recognize them early.
61+
- [**10 · Summary**]({{ '/docs/10-summary/' | relative_url }}) —
62+
Each principle, reorganized: the value it delivers, the implementation pieces that
63+
realize it, and the failure mode when it's missing.
5964

6065
## If you have time for only a few
6166

‎_docs/10-summary.md‎

Lines changed: 242 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,242 @@
1+
---
2+
title: "Summary — the four principles, realized"
3+
order: 10
4+
description: A closing summary — for each of the four data-mesh principles, the value it delivers, the implementation pieces that realize it, and what happens without it.
5+
duration: 10 min
6+
---
7+
8+
The reading set has worked through the data mesh implementation by
9+
*concern* — substrate, services, contracts, planes, mesh, scaling,
10+
observability, failure modes. This closing page reorganizes the same
11+
material by *principle*. For each of Dehghani's four principles, you'll
12+
see the value the principle delivers when it's honored, the concrete
13+
implementation pieces in this reference that realize it, and the failure
14+
mode that shows up when the principle is missing.
15+
16+
The diagrams on this page are reproduced from the implementation deck at
17+
`presentation/data-mesh-openshift/`. Because the deck pairs this reference
18+
with OpenShift specifically, the diagrams use OpenShift vocabulary in a few
19+
places — *Project* for namespace, *SCCs* for PodSecurity, *OperatorHub /
20+
OLM* for the operator install path, *AMQ Streams* for Strimzi, *CNPG* for
21+
CloudNativePG. The principles and the implementation pieces themselves are
22+
runtime-agnostic; the OpenShift framing is one concrete instance of the
23+
same shape. If your runtime is something else (vanilla Kubernetes, EKS,
24+
GKE, AKS, minikube), the substitutions are direct: read *Project* as
25+
namespace, *SCCs* as PodSecurity, *OperatorHub* as `helm install` of the
26+
relevant operators, and so on.
27+
28+
The reference's own runnable implementation lives in
29+
`examples/lgtm-datamesh/` and runs on a minikube profile — verified
30+
configuration for laptop reproducibility — but every choice it makes is
31+
recognizable on any compliant Kubernetes cluster.
32+
33+
## Principle 01 — Domain ownership
34+
35+
Each domain ships its data product on its own, with a clear owner and an
36+
enforced boundary. No central data team sits in the path between the
37+
domain that produces the data and the domains that consume it. The
38+
platform's job is to enforce the boundaries; the domain's job is to fill
39+
them with a working product.
40+
41+
The value of getting this right is that ownership is unambiguous. There's
42+
one team responsible for the customer data product, one team responsible
43+
for orders, one team responsible for inventory. Decisions about what
44+
those products look like, how they evolve, and what guarantees they
45+
provide are made by the people closest to the work.
46+
47+
![Principle 01 realized — domain ownership and the implementation pieces that deliver it]({{ '/assets/diagrams/17-value-domain-ownership.svg' | relative_url }})
48+
49+
In Kubernetes terms, the boundary is the *namespace*. Each domain gets a
50+
namespace (an OpenShift Project, on that runtime), and the platform
51+
enforces ownership through namespace-scoped RBAC (the domain team owns
52+
inside, nothing reaches across), pod security policies (PodSecurity
53+
admission on vanilla K8s, Security Context Constraints on OpenShift) that
54+
constrain what the domain's workloads may do on the node, and
55+
ResourceQuotas that prevent any single domain from starving the cluster
56+
of capacity another domain needs.
57+
58+
The reference's order, inventory, payment, shipping, and notification
59+
domains each live in their own namespace. The reference uses a single
60+
shared `capstone` namespace for the demo's clarity, but the production
61+
shape is one namespace per domain — the reference's helm chart structure
62+
supports that directly.
63+
64+
Without this principle: fuzzy boundaries and ownership vacuums. Data that
65+
nobody owns is modeled three different ways by three different teams. The
66+
mesh becomes a misnomer; it's a centralized data lake with extra steps.
67+
68+
## Principle 02 — Data as a product
69+
70+
A data product is *discoverable*, *addressable*, *trustworthy*, and
71+
*self-describing*. Consumers find it through a catalog rather than
72+
through a relationship with the producing team. They depend on it
73+
through a versioned contract rather than a phone call. They trust it
74+
because the platform — not a tribal knowledge channel — enforces the
75+
guarantees.
76+
77+
The value of getting this right is that consumption decouples from
78+
production. A consumer doesn't need to know which team owns a data
79+
product; they need to know what the product is, what it guarantees,
80+
and how to address it.
81+
82+
![Principle 02 realized — data as a product and the implementation pieces that deliver it]({{ '/assets/diagrams/17-value-data-product.svg' | relative_url }})
83+
84+
The implementation pieces that realize this in the reference: every
85+
service's Deployment + Service makes the product *addressable* — a
86+
domain consumer reaches `order-service.capstone.svc.cluster.local` and
87+
gets the order product. The Apicurio schema registry holds the
88+
*versioned contracts* — OpenAPI for REST, Protobuf for gRPC, AsyncAPI
89+
for the Kafka events — and rejects breaking changes at registration
90+
time. OpenMetadata is the catalog of *discoverability*: every product
91+
in the mesh is listed, with its owner, its schema, its lineage, and
92+
its consumption pattern. CRDs (custom resource definitions) carry
93+
*domain-specific types* declared like any other Kubernetes object, so
94+
the platform's admission policies can enforce mesh-wide standards
95+
about what a data product must look like before it's deployable.
96+
97+
The four pieces together make a product self-describing: you can
98+
discover it (catalog), depend on its shape (contract), reach it
99+
(addressable Service), and verify it conforms to mesh standards (CRD +
100+
admission).
101+
102+
Without this principle: "dumb" data products — renamed tables with no
103+
contract that can't serve themselves, govern themselves, or describe
104+
themselves to consumers. Discovery becomes a Slack channel; trust
105+
becomes word-of-mouth; governance has nowhere to live.
106+
107+
## Principle 03 — Self-serve data platform
108+
109+
Domains consume the platform's capabilities (streaming, databases,
110+
scaling, the mesh itself) *by declaration*. They don't operate the
111+
substrate. The platform team's job is to make the right things easy
112+
and the wrong things hard; the domain team's job is to declare what
113+
its product needs and let the platform deliver it.
114+
115+
The value of getting this right is that the platform stops being a
116+
bottleneck. Each domain can ship without waiting for the platform team
117+
to provision a queue, stand up a database, configure scaling, or wire
118+
in observability. The declarative pattern means the platform team
119+
scales by *adding capabilities*, not by *processing tickets*.
120+
121+
![Principle 03 realized — self-serve platform and the implementation pieces that deliver it]({{ '/assets/diagrams/17-value-self-serve.svg' | relative_url }})
122+
123+
The implementation pieces: the operator pattern (OperatorHub / OLM on
124+
OpenShift; `helm install` of upstream operators on other runtimes)
125+
makes platform capabilities *curated and lifecycle-managed* — install
126+
once, every domain consumes by declaration. Strimzi (AMQ Streams on
127+
OpenShift) makes Kafka available *as a custom resource*: a domain
128+
declares its consumer group; the operator handles the cluster.
129+
CloudNativePG (CNPG) does the same for Postgres: a domain declares its
130+
database; the operator handles the cluster, the backups, the failover.
131+
KEDA makes elastic scaling — including scale-to-zero — available as a
132+
ScaledObject, the same way: declare the demand signal, the operator
133+
handles the scaling. GitOps closes the loop by making the whole mesh
134+
*reproducible from Git*: every namespace, every operator, every
135+
ScaledObject, every Istio policy lives in a repo and is the source of
136+
truth.
137+
138+
Without this principle: every domain reinvents the same infrastructure
139+
badly. Ownership fragments into shadow platform teams — each domain
140+
builds its own Kafka cluster, its own database, its own scaling logic
141+
— and the cost of the duplication is hidden in domain budgets where
142+
nobody sees it as a platform-investment problem.
143+
144+
## Principle 04 — Federated computational governance
145+
146+
Global rules (security policy, contract-versioning rules, observability
147+
standards, data-classification rules) are enforced *by the platform*,
148+
*automatically*, *at the boundary*. Standards hold across the mesh
149+
while ownership stays decentralized. Governance isn't a review meeting
150+
after the fact; it's code that runs at the points where data products
151+
enter the system.
152+
153+
The value of getting this right is that decentralization doesn't
154+
require giving up consistency. Every domain can ship independently
155+
*because* the platform is enforcing the shared rules automatically.
156+
157+
![Principle 04 realized — federated computational governance and the implementation pieces that deliver it]({{ '/assets/diagrams/17-value-governance.svg' | relative_url }})
158+
159+
The implementation pieces: service-mesh mTLS makes every product-to-product
160+
call authenticated and encrypted, without any application code knowing
161+
about TLS. Istio's traffic-management primitives (VirtualService,
162+
DestinationRule, the canary subset routing the reference uses for the
163+
v1→v2 contract demo) make *progressive delivery* a platform capability —
164+
a contract evolves under controlled traffic, weight-shifted live, with
165+
the application layer unchanged. Kubernetes admission policies (ValidatingAdmissionPolicy
166+
on modern K8s, OPA/Gatekeeper or Kyverno on older clusters) enforce
167+
mesh-wide invariants: every data product carries an owner annotation,
168+
every namespace carries a domain label, every Deployment has its
169+
resource requests set. Prometheus/Tempo/Grafana/Kiali make governance
170+
*observable*: the metrics show the mesh's behavior, the traces show
171+
cross-product dependencies, the topology shows who's calling whom in
172+
real time. That visibility is the feedback loop that makes governance
173+
actually work — without it, governance is theatre.
174+
175+
Without this principle: governance bolted on from outside the platform
176+
never fits, or worse, re-centralizes into an approval bottleneck — the
177+
mesh's own anti-pattern of "federated governance" implemented as a
178+
review board. The
179+
[anti-patterns page]({{ '/docs/09-anti-patterns/' | relative_url }})
180+
walks through this failure mode in more detail.
181+
182+
## A trusted base for every data product
183+
184+
One more piece, beyond the four principles: the supply chain. Every
185+
data product in the mesh ships as a container image, and the platform's
186+
job extends to giving every domain a *trusted base* to layer on.
187+
188+
![A trusted base for every data product — the supply chain the platform gives each domain]({{ '/assets/diagrams/17-trusted-supply-chain.svg' | relative_url }})
189+
190+
The deck pairs this reference with OpenShift's specific supply chain:
191+
Red Hat Universal Base Image (UBI) — `ubi9/python-311` is the base for
192+
every Python service in the reference. UBI is freely redistributable
193+
and enterprise-maintained, which means a domain team gets a base image
194+
they can use without licensing constraints and that's patched on a
195+
regular cadence by Red Hat. Language dependencies come from Red Hat's
196+
verified channels or PyPI, with known provenance; the resulting images
197+
are signed and have an SBOM, and admission policies verify both before
198+
the image runs. The domain's data product layers its FastAPI / gRPC /
199+
Strawberry code on top of the trusted base — the domain owns its
200+
application; the platform owns the base.
201+
202+
On other runtimes, the supply-chain shape is the same: a curated base
203+
image (Distroless, Chainguard, Alpine, plain Debian — choose by your
204+
trust model), language dependencies from controlled channels, signed
205+
images, an SBOM at build time, and admission policies that verify both.
206+
The reference's bootstrap uses UBI because the implementation deck
207+
pairs with OpenShift; the substitution to another base is a single
208+
line in each service's Dockerfile.
209+
210+
The reason the supply chain belongs in this summary is that it's the
211+
piece of the mesh that's most often missed in discussion. Domains can
212+
own their products, products can be discoverable and self-describing,
213+
the platform can be self-serve, and governance can be federated — and
214+
none of that matters if the base images are unpatched. The mesh's
215+
weakest link is the layer underneath everyone's application code.
216+
217+
## Where to go from here
218+
219+
This reference is one worked example. The principles it implements are
220+
runtime-agnostic; the specific implementation pieces are one good
221+
choice among several reasonable choices for each. If you're starting
222+
your own data mesh on Kubernetes, the pattern is:
223+
224+
1. **Read this set straight through once** for the shape of the system,
225+
then once more for the trade-offs.
226+
2. **Run the example tree** to see the cross-product behaviors live —
227+
the trace tree across three products, the canary contract evolution
228+
under live traffic, the autoscaler reacting to a Kafka backlog.
229+
3. **Use the decision log** at `_plans/archive/capstone-decisions.md`
230+
to understand the *reasoning* for each implementation choice, not
231+
just the choice itself. The rejected alternatives are where the
232+
institutional memory of the build lives.
233+
4. **Adapt to your runtime** by swapping the substitutions: namespace
234+
for Project, PodSecurity for SCCs, `helm install` for OperatorHub.
235+
The reference's chart structure is set up to make this tractable;
236+
the bootstrap script is one of the natural places to fork.
237+
238+
The reference's most important property is that the principles and
239+
their implementation pieces are *visible together*. Conceptual material
240+
gives you the principles; vendor material gives you the pieces. This
241+
reference's job is to be the bridge — concrete enough to be runnable,
242+
explicit enough about its choices to be portable.

‎index.html‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,10 @@ <h2 class="section-heading">The set</h2>
4242
<a href="{{ '/docs/02-kubernetes-substrate/' | relative_url }}">Kubernetes substrate</a>)
4343
ground the implementation pages that follow; the
4444
<a href="{{ '/docs/09-anti-patterns/' | relative_url }}">anti-patterns</a>
45-
page closes the set with the failure modes to watch for.
45+
page surfaces the failure modes to watch for; and the closing
46+
<a href="{{ '/docs/10-summary/' | relative_url }}">summary</a>
47+
reorganizes everything by principle — what each delivers, and
48+
the implementation pieces that realize it.
4649
</p>
4750

4851
<div class="cards">
@@ -55,6 +58,8 @@ <h2 class="section-heading">The set</h2>
5558
{%- assign card_eyebrow = "Index" -%}
5659
{%- elsif doc.order == 9 -%}
5760
{%- assign card_eyebrow = "Anti-patterns" -%}
61+
{%- elsif doc.order == 10 -%}
62+
{%- assign card_eyebrow = "Summary" -%}
5863
{%- else -%}
5964
{%- assign card_eyebrow = "Page " | append: two_digit -%}
6065
{%- endif -%}

0 commit comments

Comments
 (0)