|
| 1 | +--- |
| 2 | +title: "Summary — the four principles, realized" |
| 3 | +order: 10 |
| 4 | +description: A closing summary — for each of the four data-mesh principles, the value it delivers, the implementation pieces that realize it, and what happens without it. |
| 5 | +duration: 10 min |
| 6 | +--- |
| 7 | + |
| 8 | +The reading set has worked through the data mesh implementation by |
| 9 | +*concern* — substrate, services, contracts, planes, mesh, scaling, |
| 10 | +observability, failure modes. This closing page reorganizes the same |
| 11 | +material by *principle*. For each of Dehghani's four principles, you'll |
| 12 | +see the value the principle delivers when it's honored, the concrete |
| 13 | +implementation pieces in this reference that realize it, and the failure |
| 14 | +mode that shows up when the principle is missing. |
| 15 | + |
| 16 | +The diagrams on this page are reproduced from the implementation deck at |
| 17 | +`presentation/data-mesh-openshift/`. Because the deck pairs this reference |
| 18 | +with OpenShift specifically, the diagrams use OpenShift vocabulary in a few |
| 19 | +places — *Project* for namespace, *SCCs* for PodSecurity, *OperatorHub / |
| 20 | +OLM* for the operator install path, *AMQ Streams* for Strimzi, *CNPG* for |
| 21 | +CloudNativePG. The principles and the implementation pieces themselves are |
| 22 | +runtime-agnostic; the OpenShift framing is one concrete instance of the |
| 23 | +same shape. If your runtime is something else (vanilla Kubernetes, EKS, |
| 24 | +GKE, AKS, minikube), the substitutions are direct: read *Project* as |
| 25 | +namespace, *SCCs* as PodSecurity, *OperatorHub* as `helm install` of the |
| 26 | +relevant operators, and so on. |
| 27 | + |
| 28 | +The reference's own runnable implementation lives in |
| 29 | +`examples/lgtm-datamesh/` and runs on a minikube profile — verified |
| 30 | +configuration for laptop reproducibility — but every choice it makes is |
| 31 | +recognizable on any compliant Kubernetes cluster. |
| 32 | + |
| 33 | +## Principle 01 — Domain ownership |
| 34 | + |
| 35 | +Each domain ships its data product on its own, with a clear owner and an |
| 36 | +enforced boundary. No central data team sits in the path between the |
| 37 | +domain that produces the data and the domains that consume it. The |
| 38 | +platform's job is to enforce the boundaries; the domain's job is to fill |
| 39 | +them with a working product. |
| 40 | + |
| 41 | +The value of getting this right is that ownership is unambiguous. There's |
| 42 | +one team responsible for the customer data product, one team responsible |
| 43 | +for orders, one team responsible for inventory. Decisions about what |
| 44 | +those products look like, how they evolve, and what guarantees they |
| 45 | +provide are made by the people closest to the work. |
| 46 | + |
| 47 | + |
| 48 | + |
| 49 | +In Kubernetes terms, the boundary is the *namespace*. Each domain gets a |
| 50 | +namespace (an OpenShift Project, on that runtime), and the platform |
| 51 | +enforces ownership through namespace-scoped RBAC (the domain team owns |
| 52 | +inside, nothing reaches across), pod security policies (PodSecurity |
| 53 | +admission on vanilla K8s, Security Context Constraints on OpenShift) that |
| 54 | +constrain what the domain's workloads may do on the node, and |
| 55 | +ResourceQuotas that prevent any single domain from starving the cluster |
| 56 | +of capacity another domain needs. |
| 57 | + |
| 58 | +The reference's order, inventory, payment, shipping, and notification |
| 59 | +domains each live in their own namespace. The reference uses a single |
| 60 | +shared `capstone` namespace for the demo's clarity, but the production |
| 61 | +shape is one namespace per domain — the reference's helm chart structure |
| 62 | +supports that directly. |
| 63 | + |
| 64 | +Without this principle: fuzzy boundaries and ownership vacuums. Data that |
| 65 | +nobody owns is modeled three different ways by three different teams. The |
| 66 | +mesh becomes a misnomer; it's a centralized data lake with extra steps. |
| 67 | + |
| 68 | +## Principle 02 — Data as a product |
| 69 | + |
| 70 | +A data product is *discoverable*, *addressable*, *trustworthy*, and |
| 71 | +*self-describing*. Consumers find it through a catalog rather than |
| 72 | +through a relationship with the producing team. They depend on it |
| 73 | +through a versioned contract rather than a phone call. They trust it |
| 74 | +because the platform — not a tribal knowledge channel — enforces the |
| 75 | +guarantees. |
| 76 | + |
| 77 | +The value of getting this right is that consumption decouples from |
| 78 | +production. A consumer doesn't need to know which team owns a data |
| 79 | +product; they need to know what the product is, what it guarantees, |
| 80 | +and how to address it. |
| 81 | + |
| 82 | + |
| 83 | + |
| 84 | +The implementation pieces that realize this in the reference: every |
| 85 | +service's Deployment + Service makes the product *addressable* — a |
| 86 | +domain consumer reaches `order-service.capstone.svc.cluster.local` and |
| 87 | +gets the order product. The Apicurio schema registry holds the |
| 88 | +*versioned contracts* — OpenAPI for REST, Protobuf for gRPC, AsyncAPI |
| 89 | +for the Kafka events — and rejects breaking changes at registration |
| 90 | +time. OpenMetadata is the catalog of *discoverability*: every product |
| 91 | +in the mesh is listed, with its owner, its schema, its lineage, and |
| 92 | +its consumption pattern. CRDs (custom resource definitions) carry |
| 93 | +*domain-specific types* declared like any other Kubernetes object, so |
| 94 | +the platform's admission policies can enforce mesh-wide standards |
| 95 | +about what a data product must look like before it's deployable. |
| 96 | + |
| 97 | +The four pieces together make a product self-describing: you can |
| 98 | +discover it (catalog), depend on its shape (contract), reach it |
| 99 | +(addressable Service), and verify it conforms to mesh standards (CRD + |
| 100 | +admission). |
| 101 | + |
| 102 | +Without this principle: "dumb" data products — renamed tables with no |
| 103 | +contract that can't serve themselves, govern themselves, or describe |
| 104 | +themselves to consumers. Discovery becomes a Slack channel; trust |
| 105 | +becomes word-of-mouth; governance has nowhere to live. |
| 106 | + |
| 107 | +## Principle 03 — Self-serve data platform |
| 108 | + |
| 109 | +Domains consume the platform's capabilities (streaming, databases, |
| 110 | +scaling, the mesh itself) *by declaration*. They don't operate the |
| 111 | +substrate. The platform team's job is to make the right things easy |
| 112 | +and the wrong things hard; the domain team's job is to declare what |
| 113 | +its product needs and let the platform deliver it. |
| 114 | + |
| 115 | +The value of getting this right is that the platform stops being a |
| 116 | +bottleneck. Each domain can ship without waiting for the platform team |
| 117 | +to provision a queue, stand up a database, configure scaling, or wire |
| 118 | +in observability. The declarative pattern means the platform team |
| 119 | +scales by *adding capabilities*, not by *processing tickets*. |
| 120 | + |
| 121 | + |
| 122 | + |
| 123 | +The implementation pieces: the operator pattern (OperatorHub / OLM on |
| 124 | +OpenShift; `helm install` of upstream operators on other runtimes) |
| 125 | +makes platform capabilities *curated and lifecycle-managed* — install |
| 126 | +once, every domain consumes by declaration. Strimzi (AMQ Streams on |
| 127 | +OpenShift) makes Kafka available *as a custom resource*: a domain |
| 128 | +declares its consumer group; the operator handles the cluster. |
| 129 | +CloudNativePG (CNPG) does the same for Postgres: a domain declares its |
| 130 | +database; the operator handles the cluster, the backups, the failover. |
| 131 | +KEDA makes elastic scaling — including scale-to-zero — available as a |
| 132 | +ScaledObject, the same way: declare the demand signal, the operator |
| 133 | +handles the scaling. GitOps closes the loop by making the whole mesh |
| 134 | +*reproducible from Git*: every namespace, every operator, every |
| 135 | +ScaledObject, every Istio policy lives in a repo and is the source of |
| 136 | +truth. |
| 137 | + |
| 138 | +Without this principle: every domain reinvents the same infrastructure |
| 139 | +badly. Ownership fragments into shadow platform teams — each domain |
| 140 | +builds its own Kafka cluster, its own database, its own scaling logic |
| 141 | +— and the cost of the duplication is hidden in domain budgets where |
| 142 | +nobody sees it as a platform-investment problem. |
| 143 | + |
| 144 | +## Principle 04 — Federated computational governance |
| 145 | + |
| 146 | +Global rules (security policy, contract-versioning rules, observability |
| 147 | +standards, data-classification rules) are enforced *by the platform*, |
| 148 | +*automatically*, *at the boundary*. Standards hold across the mesh |
| 149 | +while ownership stays decentralized. Governance isn't a review meeting |
| 150 | +after the fact; it's code that runs at the points where data products |
| 151 | +enter the system. |
| 152 | + |
| 153 | +The value of getting this right is that decentralization doesn't |
| 154 | +require giving up consistency. Every domain can ship independently |
| 155 | +*because* the platform is enforcing the shared rules automatically. |
| 156 | + |
| 157 | + |
| 158 | + |
| 159 | +The implementation pieces: service-mesh mTLS makes every product-to-product |
| 160 | +call authenticated and encrypted, without any application code knowing |
| 161 | +about TLS. Istio's traffic-management primitives (VirtualService, |
| 162 | +DestinationRule, the canary subset routing the reference uses for the |
| 163 | +v1→v2 contract demo) make *progressive delivery* a platform capability — |
| 164 | +a contract evolves under controlled traffic, weight-shifted live, with |
| 165 | +the application layer unchanged. Kubernetes admission policies (ValidatingAdmissionPolicy |
| 166 | +on modern K8s, OPA/Gatekeeper or Kyverno on older clusters) enforce |
| 167 | +mesh-wide invariants: every data product carries an owner annotation, |
| 168 | +every namespace carries a domain label, every Deployment has its |
| 169 | +resource requests set. Prometheus/Tempo/Grafana/Kiali make governance |
| 170 | +*observable*: the metrics show the mesh's behavior, the traces show |
| 171 | +cross-product dependencies, the topology shows who's calling whom in |
| 172 | +real time. That visibility is the feedback loop that makes governance |
| 173 | +actually work — without it, governance is theatre. |
| 174 | + |
| 175 | +Without this principle: governance bolted on from outside the platform |
| 176 | +never fits, or worse, re-centralizes into an approval bottleneck — the |
| 177 | +mesh's own anti-pattern of "federated governance" implemented as a |
| 178 | +review board. The |
| 179 | +[anti-patterns page]({{ '/docs/09-anti-patterns/' | relative_url }}) |
| 180 | +walks through this failure mode in more detail. |
| 181 | + |
| 182 | +## A trusted base for every data product |
| 183 | + |
| 184 | +One more piece, beyond the four principles: the supply chain. Every |
| 185 | +data product in the mesh ships as a container image, and the platform's |
| 186 | +job extends to giving every domain a *trusted base* to layer on. |
| 187 | + |
| 188 | + |
| 189 | + |
| 190 | +The deck pairs this reference with OpenShift's specific supply chain: |
| 191 | +Red Hat Universal Base Image (UBI) — `ubi9/python-311` is the base for |
| 192 | +every Python service in the reference. UBI is freely redistributable |
| 193 | +and enterprise-maintained, which means a domain team gets a base image |
| 194 | +they can use without licensing constraints and that's patched on a |
| 195 | +regular cadence by Red Hat. Language dependencies come from Red Hat's |
| 196 | +verified channels or PyPI, with known provenance; the resulting images |
| 197 | +are signed and have an SBOM, and admission policies verify both before |
| 198 | +the image runs. The domain's data product layers its FastAPI / gRPC / |
| 199 | +Strawberry code on top of the trusted base — the domain owns its |
| 200 | +application; the platform owns the base. |
| 201 | + |
| 202 | +On other runtimes, the supply-chain shape is the same: a curated base |
| 203 | +image (Distroless, Chainguard, Alpine, plain Debian — choose by your |
| 204 | +trust model), language dependencies from controlled channels, signed |
| 205 | +images, an SBOM at build time, and admission policies that verify both. |
| 206 | +The reference's bootstrap uses UBI because the implementation deck |
| 207 | +pairs with OpenShift; the substitution to another base is a single |
| 208 | +line in each service's Dockerfile. |
| 209 | + |
| 210 | +The reason the supply chain belongs in this summary is that it's the |
| 211 | +piece of the mesh that's most often missed in discussion. Domains can |
| 212 | +own their products, products can be discoverable and self-describing, |
| 213 | +the platform can be self-serve, and governance can be federated — and |
| 214 | +none of that matters if the base images are unpatched. The mesh's |
| 215 | +weakest link is the layer underneath everyone's application code. |
| 216 | + |
| 217 | +## Where to go from here |
| 218 | + |
| 219 | +This reference is one worked example. The principles it implements are |
| 220 | +runtime-agnostic; the specific implementation pieces are one good |
| 221 | +choice among several reasonable choices for each. If you're starting |
| 222 | +your own data mesh on Kubernetes, the pattern is: |
| 223 | + |
| 224 | +1. **Read this set straight through once** for the shape of the system, |
| 225 | + then once more for the trade-offs. |
| 226 | +2. **Run the example tree** to see the cross-product behaviors live — |
| 227 | + the trace tree across three products, the canary contract evolution |
| 228 | + under live traffic, the autoscaler reacting to a Kafka backlog. |
| 229 | +3. **Use the decision log** at `_plans/archive/capstone-decisions.md` |
| 230 | + to understand the *reasoning* for each implementation choice, not |
| 231 | + just the choice itself. The rejected alternatives are where the |
| 232 | + institutional memory of the build lives. |
| 233 | +4. **Adapt to your runtime** by swapping the substitutions: namespace |
| 234 | + for Project, PodSecurity for SCCs, `helm install` for OperatorHub. |
| 235 | + The reference's chart structure is set up to make this tractable; |
| 236 | + the bootstrap script is one of the natural places to fork. |
| 237 | + |
| 238 | +The reference's most important property is that the principles and |
| 239 | +their implementation pieces are *visible together*. Conceptual material |
| 240 | +gives you the principles; vendor material gives you the pieces. This |
| 241 | +reference's job is to be the bridge — concrete enough to be runnable, |
| 242 | +explicit enough about its choices to be portable. |
0 commit comments