Skip to content

Close the RUN-dependency tamper path; fix stale .git-protection claim; bash 3.2 postflight guard - #170

Open
CryptoJym wants to merge 3 commits into
DanMcInerney:mainfrom
CryptoJym:fix/check-dependency-surface
Open

Close the RUN-dependency tamper path; fix stale .git-protection claim; bash 3.2 postflight guard#170
CryptoJym wants to merge 3 commits into
DanMcInerney:mainfrom
CryptoJym:fix/check-dependency-surface

Conversation

@CryptoJym

Copy link
Copy Markdown

Findings from an independent audit of the loop's enforcement mechanics (I run architect-loop in production). Three text changes to the freeze/builder contract plus one script portability fix. No style changes; every finding cites the current files.

1. The freeze surface doesn't cover what the frozen checks execute

docs/checks/ is frozen and read-only (SKILL.md hard rule 2, frozen-checks freeze protocol), and postflight's touch-set diff catches out-of-bounds writes — but the files a frozen RUN command executes (test files, validators, scripts) can legitimately sit inside a job's MAY TOUCH set, and there is currently no disclosure or re-verification requirement for edits to them. A builder whose job includes test files can weaken the very test a frozen RUN item runs, and the check-runner will grade the weakened test green. Agents game the test, not the criteria file (ImpossibleBench, which DESIGN.md already cites). Field precedent: in one of my runs a builder patched a validator script mid-job to get past a scaffolding failure — its own disclosure was the only thing that caught it.

Under the danger-full-access posture (dispatch.md ## Sandbox posture) this is the primary remaining tamper path, since sandbox isolation no longer backs the prose rules.

Changes:

  • skills/frozen-checks/SKILL.md — new ## Dependency surface section: name the RUN dependency set in the check header; out of MAY TOUCH by default; in-bounds edits are disclosed and re-verified against a known-bad input before a green counts.
  • skills/to-issues/SKILL.md — decomposition keeps the linked check's RUN dependencies out of MAY TOUCH unless the issue explicitly owns them.
  • skills/architect/SKILL.md hard rule 2 — one sentence extending the freeze surface.
  • skills/architect/dispatch.md PHASE 1 — builders must record every edit to a RUN dependency under CHECK-DEPENDENCY EDITS (exact diff + reason); undisclosed = job FAIL.

2. The builder block asserts a protection that no longer exists

The block still says: "Do NOT commit - the sandbox protects .git by design" — false since the 2026-07-07 full-access directive; the real mechanism is the prose ban + postflight audit (dispatch.md ## Sandbox posture: "Builders still never commit — a prose ban audited by postflight"). A builder that discovers it can write .git has just learned the block's claims are soft — the worst possible lesson mid-job. The instruction now states the true mechanism: postflight diffs the full freeze→job range and audits for builder commits; any commit/amend/history edit fails the job.

3. PHASE 0 conflicts vs NONINTERACTIVE: when does a conflict block?

NONINTERACTIVE says "run to completion unless blocked," and PHASE 0 requires stating execution conflicts — but nothing defines when a conflict is a block, so a builder that finds the spec or check unexecutable as written will build around it and burn the job. Added one sentence: a conflict that makes the spec or a frozen check unexecutable as written stops the job as STATUS: BLOCKED with the evidence.

4. postflight.sh dies on stock macOS bash 3.2 (second commit)

On stock macOS (/bin/bash 3.2.57), set -u treats an empty array as unset, so is_allowed() aborts with exempt[@]: unbound variable — i.e. the merge gate itself errors on every job with no exemptions. tests/validate_skills.py fails on clean main here: 9 problems before the fix, 3 after (the survivors are kill-job/watchdog fixture failures that come and go across runs on macOS — flagging as an observation, not addressed here). Repro:

$ /bin/bash -c 'set -u; a=(); for x in "${a[@]}"; do :; done; echo ok'
/bin/bash: a[@]: unbound variable
$ /bin/bash -c 'set -u; a=(); for x in ${a[@]+"${a[@]}"}; do :; done; echo ok'
ok

The ${arr[@]+...} guard is the standard portable idiom; bash ≥ 4.4 behavior is unchanged.


DESIGN.md is deliberately untouched to keep the diff reviewable — happy to add the evidence entries per the maintenance rule ("No feature ships without evidence in DESIGN.md") if you want them, or feel free to take the ideas and reshape the text to house style.

🤖 Generated with Claude Code

CryptoJym and others added 2 commits July 9, 2026 16:51
…mmit rationale

Three enforcement-mechanics gaps found in an independent audit while
running the loop in production:

- Frozen checks are read-only, but the files a frozen RUN command
  executes (test files, validators, scripts) can sit inside a job's
  MAY TOUCH set and be weakened with no disclosure requirement - agents
  game the test, not the criteria file. Now: RUN dependencies are named
  in the check header, kept out of MAY TOUCH at decomposition by
  default, and any in-bounds edit must be disclosed in the job report
  (CHECK-DEPENDENCY EDITS) and re-verified against a known-bad input
  before a green counts. Under the danger-full-access posture this is
  the primary remaining tamper path.

- The builder block still told builders 'the sandbox protects .git by
  design' - false since the 2026-07-07 full-access directive (dispatch.md
  Sandbox posture: prose ban + postflight audit). A builder that
  discovers it CAN write .git learns the block's claims are soft. The
  instruction now states the real mechanism: postflight diffs the full
  freeze->job range and audits for builder commits.

- PHASE 0 names execution conflicts but NONINTERACTIVE says run to
  completion 'unless blocked' with no definition of when a conflict
  blocks. One sentence: a conflict that makes the spec or frozen check
  unexecutable as written stops the job as STATUS: BLOCKED.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On stock macOS (/bin/bash 3.2.57) 'set -u' treats an empty array as
unset, so is_allowed() dies with 'exempt[@]: unbound variable' and every
postflight fixture fails - tests/validate_skills.py fails on clean main
here (9 problems -> 3 after this fix; the remainder are nondeterministic
kill-job/watchdog fixtures). The ${arr[@]+...} guard is the standard
portable idiom; bash >= 4.4 behavior is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2fd7034ef9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread skills/architect/dispatch.md Outdated
Comment on lines +570 to +573
record the verbatim output. Do NOT commit or mutate git state - commits are
the orchestrator's alone; postflight diffs the full freeze->job range and
audits for builder commits, so any commit, amend, or history edit fails your
job regardless of results. Do NOT delete lock

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Implement the builder-commit audit before claiming enforcement

In the CLI postflight path I checked, a builder-created commit on job_branch still merges successfully when the worktree is clean: skills/architect/postflight.sh only commits dirty worktrees, then checks job_rev != freeze_rev and allowed paths before merging. I reproduced that path with a committed allowed.txt change and postflight returned POSTFLIGHT: OK, so this instruction teaches that any commit/amend/history edit fails even though the stated gate does not enforce it; either add the audit or avoid promising postflight enforcement here.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair catch, and your repro (in-bounds committed change → POSTFLIGHT: OK) confirms it — this is exactly the claims-vs-enforcement drift the PR is about, and my wording reintroduced it. Fixed in 0a4f6e9:

  • Builder block now states only the enforced consequence: postflight's touch-set diff over the freeze→job range, and the docs/checks/ auto-FAIL. The no-commit ban stands as an instruction without promising a mechanism that isn't there.
  • Both sandbox-posture lines now name the gap orchestrator-side, with the interim backstop: git log <freeze-sha>..<job-branch> before merging; builder-authored commits there are a job FAIL.

Happy to implement the authorship audit itself in postflight.sh/.ps1 plus a validator fixture as a follow-up — the one policy question I couldn't settle from outside is whether any legitimate flow pre-commits on a job branch before postflight runs (e.g. Claude Desktop isolation worktrees), which decides hard-FAIL vs typed-warn semantics.

🤖 Generated with Claude Code

Codex review reproduced a builder-authored in-bounds commit merging with
POSTFLIGHT: OK - postflight checks paths, not commit authorship. The
builder block and both sandbox-posture lines promised a commit audit that
does not exist; that is the same claims-vs-enforcement drift this PR is
about, so stop promising it:

- builder block: the no-commit ban stands as an instruction; the stated
  consequence is now the enforced one (out-of-bounds path or docs/checks/
  edit fails the job)
- sandbox posture (both spots): name the real gap orchestrator-side and
  the interim backstop - git log <freeze-sha>..<job-branch> before merge,
  builder-authored commits there are a job FAIL - until postflight grows
  an authorship audit

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant