feat!: harden the release path and gate the claims Fathom publishes - #195
Merged
Conversation
The image had not built since the Studio moved into a uv workspace member: `uv sync` parses the whole workspace, dev group included, before `--no-dev` excludes anything, so it died on `packages/fathom-studio`, which the image never copied in. Nothing caught it because no job had ever built the image. Rebuilt as a two-stage build that installs from the lock with `--frozen` and carries only the resolved virtualenv into the runtime stage, plus four fixes to things that were wrong independently of the workspace: - `mkdir -p /rules` ran after `USER fathom` and could only ever have worked against a writable `/`. Root now creates and chowns the mount point before the image drops privileges. - The CMD advertised a configurable `PORT` and then hard-coded 8080, so setting it did nothing. It is expanded at run time now, with `exec` so uvicorn stays PID 1 and signal-addressable. - Both base images float. `python:3.14-slim-bookworm` and `uv:latest` are pinned by digest, and the runtime moves to 3.13 -- the version the test matrix actually covers. Attestable images cannot re-base themselves. - `gcc` is gone: clipspy publishes manylinux wheels for every supported Python, so nothing was ever compiled from source. The new `docker` CI job builds the image and asserts the three things the docs promise: `/health` answers on a non-default `PORT`, the server runs as `fathom` and not root, and `/rules` is writable by that user. It also waits for the declared HEALTHCHECK to report healthy, so the healthcheck cannot rot into a decoration. Image is 141 MB. Signed-off-by: Sean Mauk <seanmauk@krakennetworks.com>
Publishing keyed off a tag arriving and nothing else, so 0.7.4 shipped to
PyPI while `docs` -- a required check -- was red on main. `release-gate.yml`
is a reusable workflow every publish path now calls first: it waits for the
tagged commit's check runs to finish and refuses to publish unless all of
them passed. It deliberately does not read the branch-protection contexts
list (that endpoint needs admin rights the GITHUB_TOKEN lacks), so it asserts
over every check run instead -- a superset -- with a floor so a commit that
ran nothing cannot read as success.
Alongside that:
- `workflow_dispatch: {}` is gone from both publish workflows. Neither the
`pypi` nor the `npm` environment has any protection rule or branch policy,
so it let any write-access account publish signed artifacts built from any
ref they picked.
- Signing moved into its own job. The decrypted minisign key used to sit at
/tmp/mini.key in the same job that later ran `pypa/gh-action-pypi-publish`
from `release/v1` -- a mutable BRANCH. `sign` holds the key and emits only
detached signatures; `publish` holds the PyPI identity and never sees it.
- Every action in the release path is pinned to a full commit SHA with a
version comment; dependabot tracks SHA pins and will propose bumps.
- minisign comes from its pinned upstream tarball, not apt, for the reason
the CI job just learned: one unhealthy Ubuntu mirror should not be able to
fail a release.
- npm-publish fails immediately, with the reason in the message, when
NPM_TOKEN is unset. Four of four historical runs died on `ENEEDAUTH` forty
lines into a registry error, which is why the TS SDK has never shipped.
The TS SDK also versioned independently: `package.json` sat at 0.1.0 while
the engine went to 0.8.0, so every tag asked npm to republish the same
version. release-please now writes it from the release, npm-publish refuses
to publish a package.json that disagrees with the tag, and
`check_version_sync.py` -- which the `lint` job already runs -- fails the
build on drift between any of the three version sources.
`docker-publish.yml` is new: the README advertised `kraken/fathom:latest`,
which does not exist on any registry and which no workflow had ever built.
Images now go to GHCR on tag, with build-provenance attestation, using the
built-in GITHUB_TOKEN so the release path gains no new long-lived credential.
main-health watches the three publish workflows too, and no longer filters
on `head_branch == 'main'` -- a tag push reports the tag there, so every
failed release had been silent by construction.
Signed-off-by: Sean Mauk <seanmauk@krakennetworks.com>
Dependabot's `github-actions` ecosystem writes the workflow files themselves, and both automation workflows approved and merged those PRs with no human in the loop. That is the one ecosystem where an unreviewed merge changes what CI is permitted to do -- and as of this branch the release path holds the minisign key and the PyPI identity, all of it pinned to action SHAs that such a PR rewrites. Those PRs now stay open for a person. The check is on both the `package-ecosystem` output and the branch prefix, because the output is empty on some grouped PRs and an empty string passes a `!=` test. `packages/fathom-editor` was in dependabot's auto-merge scope while having no job in any workflow, so bumps to its vite/typescript/react tree landed on main without ever being installed or built -- which is how its vite config came to fail `pnpm run build` on main unnoticed. ts-ci now builds it (`tsc && vite build`, so this typechecks too). Signed-off-by: Sean Mauk <seanmauk@krakennetworks.com>
`focus_order` ran modules in the opposite order to the one it documents. `(focus A B C)` gives the focus to A first and queues B and C behind it, but both `Compiler.compile_focus_stack` and `Evaluator._setup_focus_stack` emitted the list reversed, on the belief that a later name ended up on top of the stack. So `focus_order: [A, B]` ran B first. That is worse than a cosmetic inversion. Decisions are last-write-wins, so the module that runs last is the module whose verdict the caller receives — and the inversion handed that to whichever module the author listed *first*. `examples/02-rbac-modules` demonstrates the cost: it lists `deny_checks` first to make guardrails authoritative, and only worked because the bug ran that module last. With the ordering corrected it lists `role_permits` first, so the deny module still has the final word; without the reorder, an editor reading a confidential sheet flipped from deny to allow. No behaviour test anywhere depended on the reversal — the nine tests that failed were all string assertions over the emitted `(focus ...)` command, and `test_evaluator.py` carried a comment conceding the observed order was the opposite of what it expected before asserting only set-equality to avoid the question. It pins the order exactly now. The reference, concept and tutorial pages that documented the reversal are corrected, along with the claim that an earlier module short-circuits later ones: every listed module runs. BREAKING CHANGE: `focus_order` is now the execution order it always claimed to be. A multi-module ruleset relying on the previous inverted behaviour must reverse its `focus_order` list. Single-module rulesets are unaffected, as are rulesets whose modules do not write competing decisions. The bug surfaced while writing `tests/test_determinism.py`, which the package should have had from the start: it calls itself a "Deterministic reasoning runtime", and every test matching /determinis/ was about documentation generation. The new module asserts the three separate things determinism means — repetition, assertion-order independence, and instance independence — over both a synthetic pack built to make traces order-sensitive and the real 144-rule SSVC pack. Signed-off-by: Sean Mauk <seanmauk@krakennetworks.com>
Signed-off-by: Sean Mauk <seanmauk@krakennetworks.com>
Only clipspy and mcp had upper bounds; five requirements (prometheus_client, prometheus-fastapi-instrumentator, grpcio-tools, redis[hiredis], asyncpg) had no constraint at all. An unbounded floor lets a resolver install a future major into an environment this release was never tested against, and the break lands on the consumer's `pip install` rather than in this repo's CI. Every requirement a user can install -- the base dependencies and every extra -- now stops at the first untested major. `[dependency-groups] dev` is deliberately left unbounded: it is never installed by a consumer, and its breakage surfaces in CI, where it belongs. Also moves codespell out of the `docs` extra and into the dev group. codespell is GPL-2.0-only and this package is MIT; shipping it in a user-installable extra put a copyleft tool one `pip install fathom-rules[docs]` away from a user's environment. Nothing breaks: docs.yml runs `uv sync --extra docs`, which installs the dev group too. Relocking changed no resolved version -- the diff is metadata only.
README published four performance targets and nothing in the repo measured them, so they were true only by accident. One of them was not true at all: "YAML compilation < 50ms" held for a 100-rule synthetic pack and was already false for the SSVC pack this repo ships, which takes ~164ms for its 144 rules. A flat millisecond budget is a claim about pack size rather than about the compiler. scripts/benchmark.py now holds the numbers and measures all four operations; the new `bench` job runs it on every pull request and fails on a regression. README is restated to match what is actually measured: fact assertion moves to < 25µs (measured 9µs, the old 10µs bar had no headroom for a shared runner) and compilation becomes < 2ms per rule, which both the synthetic pack (0.5ms/rule) and SSVC (1.1ms/rule) meet. tests/test_scripts/test_benchmark.py fails if the table and the script ever disagree. Gating is on medians, not maxima: one sample on a shared runner can be an order of magnitude off for reasons unrelated to this code, and a max-based gate on a 25µs budget would be red most mornings. The p95 is printed so a real regression stays visible before it crosses the bar. Also adds a `dependency-review` job. A bump that introduces a known-vulnerable package is otherwise invisible until Dependabot alerts on it, which is after it has landed on main; this diffs the manifest in the PR and fails on a high-severity advisory. Licences are advisory only -- a copyleft tool in a dev group is a judgement call, not a build break.
README.md and docs/index.md both announced 0.7.0 while 0.7.4 was on PyPI, and neither could be caught: check_version_sync.py only compared pyproject.toml, __init__.py and the TS SDK manifest. Those two lines are the first version a reader sees on GitHub and on the docs site. Both lines now carry an `x-release-please-version` marker and are listed as release-please extra-files, so the release PR rewrites them, and the sync check reads them too, so a dropped marker or a reworded line fails loudly instead of drifting silently. A prose file whose version line no longer matches is an error rather than a skip -- silently skipping is how a rewritten README would disable the gate it just broke. Also brings both lines up to the released 0.8.0.
…d client Two copies of the API contract existed. The one at the repo root was frozen at API version 0.3.0 from April 2026 and had never been regenerated; the live one, docs/reference/rest/openapi.json, is written by scripts/export_openapi.py, is at 0.8.0, and is held to the running app by a test that fails when it drifts. The root copy was the input both SDK generators pointed at, so both were wired to a spec missing /v1/rules/reload and /v1/status. packages/fathom-ts/src/generated/ was the output of that stale spec. It was never imported: src/index.ts exports only the hand-written client and error types, and nothing in src/ references it. It could not be refreshed either -- `openapi-ts -c fetch` emits zero files against the pinned @hey-api/openapi-ts 0.98, because a bare client name is no longer valid there. Rather than commit a second unused tree in the new layout, the tree and its generate script go; the SDK is hand-written and now says so. packages/fathom-go/Makefile's OPENAPI_SPEC default moves to the live spec, so its (opt-in, never-committed) `make generate` target no longer points at a file that does not exist. docs/reference/planned-integrations.md claimed the generated client "has been produced and committed" and that the TS suite had no CI; both were false. It now states the endpoint coverage the hand-written client actually has -- 4 of the 10 documented endpoints -- and README drops the "OpenAPI-generated client pending" line for the same reason.
First run on a GitHub runner came back at 1841µs/rule median for the SSVC compilation case against a 2000µs/rule bar, with a p95 of 2003 -- green, but only just, and a gate that lands 8% under its limit is a flake generator rather than a regression detector. The runner is ~1.7x slower than the machine the published numbers describe (1106µs/rule locally for the same case). Rather than weaken what README promises a user holding a laptop, `--slack` multiplies every limit before comparing and CI passes 1.5. The printed table names the factor, the error message quotes both the limit and the published target, and a slack below 1.0 is refused so the flag can only ever loosen -- never quietly gate tighter than the claim. README says which number is which: the table is a developer machine, CI enforces it with the allowance.
… entry CHANGELOG.md stopped at 0.3.0 while 0.8.0 was on PyPI, and its `[Unreleased]` section described a breaking change -- the removal of `FunctionDefinition.type = "temporal"` -- that had shipped in 0.3.3 three months earlier. pyproject.toml points every PyPI visitor at this file, so it was telling users a released breaking change was still pending. Backfills 0.3.1 through 0.8.0 as curated entries, with the `[Unreleased]` block moved under 0.3.3 where `git tag --contains` puts it. A "Release history notes" section records the three gaps the tag list does not explain on its own: 0.3.2 is on PyPI with no tag, 0.3.3 and 0.4.0 are tagged but never reached PyPI (0.4.0's tag is `fathom-rules-v0.4.0`, which `pypi-publish.yml`'s `v*.*.*` filter does not match), and signatures exist only from 0.5.0 onward. release-please runs with `skip-changelog`, so nothing was ever going to fill this in on its own. check_version_sync.py now also requires the newest CHANGELOG heading to be the version in pyproject.toml, which turns a missing entry into a red release PR instead of a silent gap. An `[Unreleased]` heading does not satisfy it. CONTRIBUTING documents the one manual step this adds to a release.
The page opened with "Every Fathom release artifact published to PyPI and attached to a GitHub Release is signed", with no version floor anywhere in it. Five of the thirteen PyPI releases (0.1.0 through 0.3.2) have no signature, and the GitHub releases for 0.3.0, 0.3.1, 0.3.3 and 0.4.0 have no assets at all -- a reader following this page for one of those versions finds nothing to verify and cannot tell whether that is a supply-chain problem or a history problem. States the floor (0.5.0), lists exactly which releases are unsigned, and says plainly that nothing will be back-signed: producing a signature today for an artifact built in April would assert a custody chain that did not exist at build time. Also records the enforcement that does exist -- pypi-publish.yml fails before upload if any artifact lacks its .minisig.
se-jo-ma
changed the title
se-jo-ma
marked this pull request as ready for review
August 19, 2026 22:35
… prose The admonition's indented body tripped MD046 (code-block-style) in the docs-quality job, and no other page in docs/ uses admonition syntax.
krakennetworks
approved these changes
Aug 20, 2026
This was referenced Aug 20, 2026
se-jo-ma
pushed a commit
that referenced
this pull request
Aug 20, 2026
The `focus_order` fix in #195 carried a `BREAKING CHANGE:` footer, and release-please's default for a 0.x project is to treat that as the signal to cut 1.0.0 -- so PR #196 opened as "chore(main): release 1.0.0". Nothing about that branch was a decision to declare the API stable. `bump-minor-pre-major` makes a breaking change bump the minor while the project is on 0.x, which is what SemVer allows pre-1.0 and what VERSIONING.md now documents. `bump-patch-for-minor-pre-major` stays off: a feature should still move the minor, not hide in a patch. 1.0.0 becomes a deliberate act -- a commit that raises the version explicitly, after the readiness audit -- rather than a side effect of a commit footer. Signed-off-by: Sean Mauk <seanmauk@krakennetworks.com>
se-jo-ma
pushed a commit
that referenced
this pull request
Aug 20, 2026
The `focus_order` fix in #195 carried a `BREAKING CHANGE:` footer, and release-please's default for a 0.x project is to treat that as the signal to cut 1.0.0 -- so PR #196 opened as "chore(main): release 1.0.0". Nothing about that branch was a decision to declare the API stable. `bump-minor-pre-major` makes a breaking change bump the minor while the project is on 0.x, which is what SemVer allows pre-1.0 and what VERSIONING.md now documents. `bump-patch-for-minor-pre-major` stays off: a feature should still move the minor, not hide in a patch. 1.0.0 becomes a deliberate act -- a commit that raises the version explicitly, after the readiness audit -- rather than a side effect of a commit footer. Signed-off-by: Sean Mauk <seanmauk@krakennetworks.com>
se-jo-ma
pushed a commit
that referenced
this pull request
Aug 20, 2026
A pack that declared modules but no `focus_order:` fired nothing at all. CLIPS drains only the agenda of the module holding the focus, so with an empty focus list every rule scoped to a declared module sat unfired and the caller got the default decision back — a wrong answer, not an error, from a pack whose rules all matched. `load_modules` now focuses the declared modules in declaration order when nothing else has set a focus, and only then: a `focus_order:` in any file or an earlier `set_focus` still wins outright. This invalidated the benchmark added in #195, which generated exactly such a pack: its two evaluation cases were timing an empty agenda and reporting healthy microseconds for doing nothing. The generated pack now declares its focus, and the script refuses to report timings for a pack whose rule_trace comes back empty, so the same mistake cannot pass silently again. The real numbers are inside their published targets (single 71us, 100-rule 156us). Docs: this behaviour is now stated in the module reference, the runtime concept page, and the modules tutorial. The rest of the pages here are the source-drift backlog from #195, which the docs job only warns about: - writing-rules.md taught YAML that no longer compiles — bare `expression: active` (must be `equals(active)`) and `alias: req` (must be `$req`), plus an emission claim the compiler never made. - template.md said no validator was applied to SlotDefinition.name, allowed_values, or default; all three are validated now. - audit-attestation.md was missing the input-fact snapshot step and had a stale `sign()` signature. - yaml-compilation.md said module `priority` drives focus ordering. - function.md's validator table was missing the hierarchy validators and the unknown-key rejection; cli.md did not mention that `compile` now emits literals by declared slot type; hot-reload.md told readers to use a reload listener without naming `Engine.subscribe_reload`. - fact.md and five-primitives.md carried line citations that had moved. Signed-off-by: Sean Mauk <seanmauk@krakennetworks.com>
se-jo-ma
pushed a commit
that referenced
this pull request
Aug 20, 2026
The doc-source freshness gate was advisory in every job that ran it, and the result is the previous commit: eight reference pages describing validators, YAML forms, and a call signature that had not existed since #195, one of them teaching rule YAML that no longer compiles. A warning nobody reads is not a gate. It was advisory for a real reason, though: editing a source ages every page that cites it, so a whole-repo gate fails whoever opens the next pull request rather than the author who changed the source. That is the treadmill that got it switched off, and turning it back on unchanged would just rebuild it. `--changed-vs REF` narrows the check to the sources the branch itself touched. Pages citing files this branch did not edit are someone else's drift and stay in the advisory sweep; pages citing files it did edit must be re-verified here. An unreadable ref falls back to checking everything rather than passing, since a gate that no-ops on a bad ref is worse than a noisy one. The required `docs` job now runs the scoped check on pull requests, and CONTRIBUTING documents what a contributor has to do when it fires. Signed-off-by: Sean Mauk <seanmauk@krakennetworks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Milestone 0.9: the release path, the gates around it, and the claims the project publishes about itself.
Release path
/ruleswritable, honours$PORT, declaredHEALTHCHECK— and adockerjob that builds it and asserts all four on every PR.release-gate.ymlis a reusable workflow that polls the tag's check runs and refuses to publish unless every one has completed successfully, withlintanddocsexplicitly required to have run.pypi-publish,npm-publishand the newdocker-publishall call it first.workflow_dispatchis gone from the PyPI path, every action is SHA-pinned, signing runs in its own job so the minisign key never shares a runner with the publish step, and every artifact must have a.minisigbefore upload.main.Gates on published claims
scripts/benchmark.pynow holds the numbers, thebenchjob enforces them on every PR, and a parity test fails if README and the script disagree. Fact assertion is restated at < 25µs and compilation at < 2ms per rule, both measured. CI applies a documented--slack 1.5because a shared runner is ~1.7× slower than the machine the targets describe.docs/index.mdsaid 0.7.0 while 0.7.4 was on PyPI. Both lines are now release-pleaseextra-filesand covered bycheck_version_sync.py.CHANGELOG.mdstopped at 0.3.0 and described an already-shipped breaking change as[Unreleased]. Backfilled 0.3.1 → 0.8.0, with a notes section recording the tag/PyPI gaps. A missing entry for the version inpyproject.tomlnow failslint.Correctness
focus_orderran modules backwards.(focus A B C)gives A the focus first, but both the compiler and the evaluator emitted the list reversed, sofocus_order: [A, B]executed B first — and since the evaluator is last-write-wins, the module listed first decided the outcome. Fixed in both places, with tests, docs and the RBAC example updated.tests/test_determinism.pynow asserts the determinism claim directly: repetition, assertion-order independence, and instance independence, on a synthetic pack and on SSVC.Dependency and SDK hygiene
codespell(GPL-2.0-only) moves out of the user-installabledocsextra into the dev group.openapi.jsonwas frozen at API 0.3.0 and was the input both SDK generators pointed at. Deleted, along withpackages/fathom-ts/src/generated/— a tree nothing imported, generated from that stale spec by a command that emits zero files against the pinned@hey-api/openapi-ts.github_actionsbumps (those PRs rewrite the release path itself), andfathom-editorfinally has a build job.Still needs a human
CodeQLtomain's required status checks (branch-protection API call was blocked from this session).NPM_TOKENon thenpmenvironment —npm-publishhas never succeeded without it.pypiandnpmenvironments (both currently have none).fathom-rules-v0.4.0does not matchpypi-publish.yml'sv*.*.*filter, and PyPI 0.3.2 has no tag. Both are documented in CHANGELOG for now.docker,bench,editor-buildanddependency-reviewrequired contexts.BREAKING CHANGE:
focus_ordernow executes modules in the order they are written. A ruleset that relied on the previous reversed order — where the module listed first produced the caller-visible decision — must swap itsfocus_orderlist.