Skip to content

Use JSON schema to validate incoming tasks - #212

Open
kevinchannon wants to merge 69 commits into
mainfrom
schema-validation-pipeline
Open

Use JSON schema to validate incoming tasks#212
kevinchannon wants to merge 69 commits into
mainfrom
schema-validation-pipeline

Conversation

@kevinchannon

Copy link
Copy Markdown
Owner

This PR is being opened prematurely to allow assessment of the state of CI tests against this work.

Eventually thë whole feature will be present and we can merge it.

@claude

claude Bot commented Jul 3, 2026

Copy link
Copy Markdown

Review

This PR is explicitly a WIP checkpoint (slices 0–3 of the schema-validation-pipeline plan, issue #43), so I focused the review on the code that's actually landed rather than on the plan's not-yet-built slices.

Summary of the change

  • Unifies runner/interpreter field substitution onto the Jinja renderer (Executor._render_runner_field), deleting the old regex-based _substitute_builtin/_substitute_env wrappers in executor.py.
  • Adds a generic, regex-based reference walker (src/tasktree/template_refs.py: collect_template_refs, expand_variable_refs) for discovering {{ prefix.name }} references across a raw recipe subtree.
  • Adds check_runner_template_refs (parser.py), called from build_recipe_runner and _parse_inline_interpreter, which rejects arg.*/dep.*/self.* and per-task tt.* builtins (task_name, working_dir, timestamp, timestamp_unix) inside runner/interpreter definitions at parse time, since those definitions are shared across tasks and rendered once.
  • task_config.build_runner_config assembles the restricted (env + tt only) rendering context for runner fields.

Code quality / best practices

  • Nice adherence to the project's stated philosophy: small commits, tests alongside each increment, and a documented "reference gate" comparing behaviour against v1.3.2 with an explicit expected-divergences list. The plan doc is kept in sync with what actually landed.
  • check_runner_template_refs and build_runner_config are small, well-named, single-purpose functions with docstrings that state pre/post-conditions — matches the repo's "small named functions over comments" guidance.
  • Minor redundancy: build_recipe_runner calls check_runner_template_refs(config, ...) on the entire raw runner config (which already includes a nested interpreter: subtree), and then parse_interpreter_spec_parse_inline_interpreter calls it again on just the interpreter subtree. Harmless (recipes are small, parsed once), but worth a one-line comment noting it's intentional double coverage, or a follow-up to avoid re-walking the same nodes.
  • The plan doc (docs/plans/schema-validation-pipeline.md) keeps each "✅ done" slice's completion summary directly above the original (now historical) planning prose for that slice ("Original scope follows."). Intentional per the doc's own convention, but it reads as duplicated content on a first pass — worth a note for future readers unfamiliar with the convention.
  • The plan records that "Kevin has granted per-increment local commits for this work (supersedes CLAUDE.md's never-commit rule)" — good that the exception is explicit and documented in the plan rather than silently diverging from CLAUDE.md.

Potential bugs / issues

  • template_refs._REFERENCE's name pattern ([A-Za-z_][A-Za-z0-9_-]*(?:\.[A-Za-z_][A-Za-z0-9_-]*)*) requires each dotted segment to start with a letter/underscore, so a positional reference like {{ self.inputs.0 }} only captures self.inputs, dropping the numeric index. This doesn't affect the current use (check_runner_template_refs only needs to know a forbidden prefix was used, not the exact dotted path), but it's a latent under-matching gap if collect_template_refs is later reused for hashing/pruning (per the plan's slice 5 notes), where the full dotted path matters more. Worth a test/fix before that reuse happens.
  • Defense-in-depth is solid: even if the regex walker under-matches (e.g. bracket-style Jinja access like {{ arg['x'] }}), the restricted build_runner_config context (no arg/dep/self keys) plus StrictUndefined means such a reference would still fail at render time — just with a less specific error message than the parse-time check produces. Not a correctness bug, just a slightly worse error message in that edge case.

Performance

  • No concerns. The added regex walking runs once at parse time over small YAML trees; no hot-path impact.

Security

  • No concerns. This is pure validation/rendering logic operating on trusted recipe YAML — no new subprocess/eval/network surface, and the regexes have no catastrophic-backtracking shape (no nested quantifiers).

Test coverage

  • Strong coverage for what's landed: tests/unit/test_template_refs.py covers the walker (all prefixes, nested dicts/lists/keys, Jinja conditionals, cycles/fixpoint closure); tests/unit/test_runner_template_restriction.py covers end-to-end parse-time rejection/acceptance across runners, inline task runners, and interpreters, and is deliberately kept import-minimal so it doubles as the v1.3.2 reference-gate fixture; tests/unit/test_executor.py adds parity + rejection tests per runner field (volumes, env_vars, ports, working_dir, run args, preamble).
  • One small regression in assertion strength: tests/integration/test_builtin_variables.py's TASK_NAME_VARUSER_NAME_VAR swap changed the assertion from checking an exact expected value ("docker-test") to just assertNotIn("{{", ...). That's a reasonable trade-off since tt.user_name is environment-dependent, but it's worth knowing the test now only proves "something got substituted," not "the right thing got substituted."

Action items

  • Consider avoiding the double check_runner_template_refs walk (once over the whole runner config in build_recipe_runner, again over just the interpreter subtree in _parse_inline_interpreter) — e.g. by having build_recipe_runner skip the interpreter key before checking, or documenting why the double-check is intentional.
  • Add a test (or fix) for positional self.inputs.N/self.outputs.N references in collect_template_refs, since the current regex drops the trailing numeric segment — flag this now so it doesn't silently bite slice 5's planned reuse of the walker for reachability/pruning.
  • Optional: strengthen the updated USER_NAME_VAR integration test (e.g. mock/patch user_name to a known value) so it verifies the correct value was substituted, not just that substitution happened.

kevinchannon added a commit that referenced this pull request Jul 4, 2026
- collect_template_refs keeps numeric continuation segments, so
  positional references like {{ self.inputs.0 }} retain their index
  (mattered for slice 5/7 reuse, not the current prefix check).
- The double check_runner_template_refs walk (whole runner config in
  build_recipe_runner, interpreter subtree again in
  _parse_inline_interpreter) is documented as intentional.
- test_builtin_vars_in_runner_volumes pins os.getlogin and asserts the
  exact substituted USER_NAME_VAR value. Reference gate: passes on
  v1.3.2 (parity).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@kevinchannon

Copy link
Copy Markdown
Owner Author

Review action items addressed in d2b44e4:

  1. Double check_runner_template_refs walk — kept, now documented as intentional at the call site in build_recipe_runner. The outer walk covers the whole raw config; the inner check in _parse_inline_interpreter is what protects interpreters defined outside any runner (the interpreters: section and task-level overrides). Skipping the interpreter key in the outer walk would couple the function to that call graph for no measurable saving.

  2. Numeric segments dropped from positional refs — fixed. _REFERENCE continuation segments may now start with digits, so {{ self.inputs.0 }} extracts inputs.0 (and {{ dep.build.outputs.0 }} keeps its index). Tests added for both. As the review noted, this didn't affect the current prefix-only consumer, but it closes the under-matching gap before slice 5/7 reuse the walker for pruning/hashing.

  3. USER_NAME_VAR assertion strength — strengthened. The test now pins os.getlogin to a known value and asserts the exact substituted string, restoring the pre-change assertion strength without depending on the environment. Reference gate: the updated test passes against v1.3.2 (parity).

Full pyramid run after the changes: unit+integration 1647 passed / 4 conditional skips; e2e 71 passed / 5 conditional skips.

@claude

claude Bot commented Jul 4, 2026

Copy link
Copy Markdown

Review of PR #212 — "Use JSON schema to validate incoming tasks" (slices 0-3, WIP)

Thanks for the clear per-slice commit history and the self-documenting plan doc - it made this easy to follow even without the originating conversation. This is explicitly an in-progress WIP PR per the description, so I focused on the substance of slices 1-3 that has landed rather than the overall roadmap.

Summary of what's in this PR

  • Slice 1: runner/interpreter field substitution (volumes, ports, env_vars, working_dir, dockerfile, context, docker build/run args, interpreter cmd/preamble) is unified onto the Jinja rendering.render() path via a new Executor._render_runner_field, replacing the old regex-based _substitute_builtin/_substitute_env helpers (deleted).
  • Slice 2: a new, generic collect_template_refs/expand_variable_refs walker in template_refs.py that finds {{ prefix.name }} references anywhere in a raw recipe subtree.
  • Slice 3: check_runner_template_refs (parser.py) uses the slice-2 walker to reject arg.*/dep.*/self.* and per-task tt.* builtins (task_name, working_dir, timestamp, timestamp_unix) inside runner/interpreter definitions, since runners are shared across tasks and rendered once.

Correctness issue: the "runners can't see per-task values" guarantee doesn't actually hold everywhere

The core safety property this slice is building towards is that runner/interpreter fields must not depend on per-task state, because a runner is shared across tasks. Two gaps undermine that today:

  1. Machine-config runners silently pick up per-task tt.* values instead of failing. check_runner_template_refs is only wired into the recipe path (build_recipe_runner in parser.py:2096, and _parse_inline_interpreter for interpreter definitions). config.py's default-runner loader (~line 298) builds machine-config runners straight through runner_from_config, which never calls check_runner_template_refs. The plan doc (docs/plans/schema-validation-pipeline.md, slice 3 notes) states this is fine because "their forbidden refs still fail at render via slice 1's strict Jinja" - but that's only true for arg/dep/self (absent from the render context entirely, so Jinja's StrictUndefined raises). It is not true for tt.*: task_config.build_runner_config (task_config.py:173-176) does "tt": dict(builtins or {}), forwarding the entire builtins dict - including task_name, working_dir, timestamp, timestamp_unix - unfiltered into the Jinja context (confirmed via Executor._collect_builtin_variables, executor.py:239-297, which always includes those four per-task keys, and _run_task_in_docker, executor.py:1399, which passes that dict straight into _substitute_builtin_in_runner/_render_runner_field). So a machine-config runner with e.g. env_vars: {TASK: "{{ tt.task_name }}"} will render successfully - silently varying its value depending on which task happens to be running - instead of erroring, which is exactly the class of bug this feature is meant to close off. There's no test exercising this path (grep -rn "runner_from_config" tests/ turns up nothing checking tt.task_name/per-task rejection there).
  2. Recipe runners can bypass the check via variable indirection. check_runner_template_refs calls collect_template_refs(subtree) directly on the raw config - it never calls expand_variable_refs (the transitive-closure helper slice 2 built specifically for this kind of chasing). So runners: { docker: { env_vars: { X: "{{ var.mount }}" } } } with variables: { mount: "{{ arg.thing }}" } passes the parse-time check (it only sees var.mount), and only fails later at render time with a generic "Undefined variable: 'arg' is undefined" Jinja error instead of the intended, more actionable message pointing at the runner/interpreter definition. Minor by comparison to (1) since it does still fail, just with a worse error and later than intended.

Suggest: for (1), either call check_runner_template_refs from config.py's runner-building path too, or (better, since it closes the hole for both paths at once) filter build_runner_config's tt dict down to the allowed four global names always, so silently-wrong values become fail-fast even for uncovered call sites. For (2), run the check against expand_variable_refs(collect_template_refs(subtree), raw_variables) rather than the raw refs, so indirection through a variable doesn't hide a forbidden namespace.

Smaller notes

  • docs/plans/schema-validation-pipeline.md's slice-3 completion note asserting per-task tt.* refs "fail at render" for machine-config runners should be corrected/removed once (1) above is addressed - right now it documents behaviour that isn't actually true, which could mislead whoever picks up slice 4+.
  • Good instinct writing tests/unit/test_runner_template_restriction.py self-contained (only importing parse_recipe) for the v1.3.2 reference-gate comparisons - matches the plan's own stated gate practicalities.
  • Test coverage for the new Jinja-based runner-field rendering (slice 1) and the walker (slice 2) both look solid and well-targeted at the field list actually in play (including the wider list found in slice 0's extra finding: dockerfile/context/build+run args/interpreter cmd+preamble).

Performance / security

No concerns - collect_template_refs's regex is non-greedy and bounded per {{ ... }} block, no ReDoS risk from the patterns used. No injection concerns; this is parse-time validation logic only.


Action items

  • Close the machine-config runner gap: either call check_runner_template_refs from config.py's runner-loading path, or make build_runner_config's tt namespace always restrict to the four global builtins (project_root, recipe_dir, user_home, user_name) regardless of caller, so per-task values can never silently leak into a shared runner's rendered fields.
  • Add a regression test that a machine-config runner referencing {{ tt.task_name }} (or another per-task builtin) either fails to parse/load or fails to render - currently nothing catches this.
  • Have check_runner_template_refs use expand_variable_refs (not just collect_template_refs) so a forbidden namespace referenced indirectly through a {{ var.name }} in a runner/interpreter field is caught at parse time with the intended error message, not a generic Jinja "undefined" error at render time.
  • Update the "forbidden refs still fail at render via slice 1's strict Jinja" note in docs/plans/schema-validation-pipeline.md once the above is fixed, since it currently overstates what's actually guaranteed for tt.* in machine-config runners.

kevinchannon and others added 26 commits August 2, 2026 22:59
Both verifications pass: the Jinja renderer covers every namespace the
unified path needs, and no fixture or test uses arg./dep./self. templates
in runner or interpreter fields. Slice 1's field list is expanded to match
the regex path's true coverage (dockerfile, context, build/run args and
interpreter cmd/preamble, in addition to the originally listed fields).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Volumes now go through rendering.render with a runner-scoped context
(env + tt only, via the new build_runner_config) instead of the chained
regex substitutions. Per-task namespaces (arg/dep/self) in a volume are
now an error instead of being silently left in the mount string.

Reference gate: parity test passes on v1.3.2; the arg-rejection test
fails there as expected (documented in the plan's divergences list).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same move as volumes: env_vars values go through the runner-scoped
Jinja context (env + tt), so per-task namespaces in an env_vars value
now error instead of silently reaching the container as literal text.

Reference gate: parity test passes on v1.3.2; arg-rejection test fails
there as expected (covered by the documented divergence).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same move as volumes and env_vars. Reference gate: parity passes on
v1.3.2; arg-rejection fails there as expected (documented divergence).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same move as volumes/env_vars/ports. Reference gate: parity passes on
v1.3.2; arg-rejection fails there as expected (documented divergence).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The shared subst() closure now uses the runner-scoped Jinja render,
migrating docker build/run args, interpreter cmd/preamble, dockerfile
and context in one move. Pre-existing parity tests for these fields
pass unchanged; new rejection tests pin the arg.* error behaviour.

Reference gate: parity via the existing (unchanged) tests; both new
rejection tests fail on v1.3.2 as expected (documented divergence).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_substitute_builtin and _substitute_env have no callers now that all
runner fields render through the Jinja engine. The underlying
substitution.py functions remain (still used on the parse-time path).

No behaviour change, so no reference-gate run (no behaviour-level test
added or changed).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Slice 2 of the schema validation pipeline (issue #43): the generic
reference walker, starting with regex extraction from a single string.
Purely additive - nothing calls it yet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dict keys are walked too, per the over-matching bias: a template in a
key position (e.g. env_vars) must not be silently missed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Chases referenced variables' definitions to a fixpoint so consumers see
everything a subtree transitively depends on. Undefined names are kept
but not chased - discovery is not validation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The generic walker already sees templates inside eval commands, read
paths and env defaults; the { env: NAME } form needs a special case
because it names its env var as a bare string. Cyclic definitions
terminate via the chased set.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Slice 3 of the schema validation pipeline (decision 4): runners and
interpreters are shared across tasks and render once, so arg/dep/self
and per-task tt builtins may not appear in their definitions. The check
uses the slice-2 walker; nothing calls it yet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
build_recipe_runner now runs check_runner_template_refs on the raw
config, covering section runners and inline task runners. The
builtin_vars_runner_volumes fixture drops its tt.task_name env var
(now-forbidden per-task builtin) in favour of tt.user_name.

Reference gate (v1.3.2): the four rejection tests fail there with
'ValueError not raised' - the recipes were silently accepted - matching
the expected-divergences entries; the per-task tt divergence is added
to the list. The acceptance test and the updated volumes test pass on
the reference.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_parse_inline_interpreter runs check_runner_template_refs, covering the
interpreters section, task-level inline interpreters and runner
interpreter fields. Error wording generalised to name interpreters.

Reference gate (v1.3.2): all six rejection tests fail there with
'ValueError not raised' (recipes silently accepted), matching the
expected-divergences entries; both acceptance tests pass there.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Status header updated (slices 0-3 done), the full-pyramid test bar and
per-increment commit permission recorded in process decisions, gate
practicalities from slices 2-3 added to the reference-arbiter section,
and slice 4 gets the parser entry points surveyed this session.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- collect_template_refs keeps numeric continuation segments, so
  positional references like {{ self.inputs.0 }} retain their index
  (mattered for slice 5/7 reuse, not the current prefix check).
- The double check_runner_template_refs walk (whole runner config in
  build_recipe_runner, interpreter subtree again in
  _parse_inline_interpreter) is documented as intentional.
- test_builtin_vars_in_runner_volumes pins os.getlogin and asserts the
  exact substituted USER_NAME_VAR value. Reference gate: passes on
  v1.3.2 (parity).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CircularImportError moves to raw_merge (its natural home once the merge
phase owns imports); parser re-exports it so existing imports keep working.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same rules as _parse_file: undotted deps get the namespace prefix; dotted
deps are prefixed only when their root segment is one of the importing
file's own import namespaces, otherwise kept as absolute references.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mirrors _parse_file: an imported file's runners come along only when a
pinned task references them; imported 'default' declarations are dropped.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Var refs are rewritten by a generic tree walk over each imported file's
local content (values only), deliberately broader than the old per-field
rewrite: dep argument templates and inline definitions are covered too.
VAR_REFERENCE_REWRITE_PATTERN moves to raw_merge; parser imports it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
kevinchannon and others added 10 commits August 2, 2026 22:59
parse_recipe gains prune_unreachable (opt-in from execute_dynamic_task
only): when the root task exists, unreachable tasks are dropped from the
merged tree before construction, so their defects are tolerated.
--list/--show/--tree leave the flag off and still validate the whole
file (--show/--tree pass a root task for lazy variable evaluation, so
pruning cannot key off root_task alone). defined_task_names now captures
the pre-pruning universe for name-aware state pruning.

Gate: both tolerance tests fail on v1.3.2 as intended (it validated
every task); the three validation-coverage parity tests pass there.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
prune_unreferenced_runners/_interpreters (raw_merge) run right after
task pruning: the 'default' declarations and their targets always
survive, task/runner {use:} references are chased from survivors only,
and the CLI --runner/--interpreter override names are threaded through
parse_recipe as keep-hints so overrides keep working. Listing/showing
paths still build and validate every definition.

Gate: the four tolerance tests fail on v1.3.2 as intended; the six
override/coverage parity tests pass there.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
evaluate_variables now derives the reachable set with
collect_reachable_task_names on the merged tree and discovers referenced
variables with collect_template_refs over the reachable tasks' subtrees
plus their referenced runners (and the default runner). The object-based
collect_reachable_tasks / collect_reachable_variables and their
enumerated field list are deleted.

Gate: parity (unreachable tasks' variables stay unevaluated) passes on
v1.3.2; the coverage divergence test fails there with 'Variable not
defined' - the old enumerated list only looked inside Docker runner
definitions, so lazy parsing broke on vars referenced only by host
runner fields. Divergence recorded in the plan.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Also repurposes TestParityWithObjectPath as a merged-tree consistency
check (its cutover parity role is obsolete now parse_recipe builds on
the merge).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The variables value schema listed 'integer' and 'number' as separate
oneOf branches, so every integer matched both and failed validation --
'variables: {port: 8080}' was rejected outright (4 fixtures in this repo
hit it). 'number' covers integers, so the redundant branch goes.

Found by auditing the file schema against what the parser accepts, ahead
of wiring the schema in at runtime.
A runner definition carried a 'default: true' key. The parser has no
whitelist of runner keys, so it was silently ignored -- the default
runner is declared as 'runners: {default: <name>}' at section level, not
inside a definition. The schema (additionalProperties: false) rightly
rejects it, so it would break once validation runs at parse time.
355 recipes under tests/fixtures are exactly the recipes tt is expected
to handle, so they are the sharpest available check that the file schema
is not stricter than the parser -- the property slice 6 needs before the
schema starts rejecting recipes at runtime. Rejections are compared
against an explicit allowlist of fixtures the schema is meant to reject,
so a new deliberate negative forces a conscious entry rather than
quietly widening the tolerated set.

Checked the net bites: artificially narrowing the variables schema to
strings turns the expected 1 rejection into 26.
Plan decision 1 assumed users legitimately write templates in typed
fields, so the schema would have to accept 'native type OR {{...}}'.
Probing the parser says otherwise: min/max, arg 'type', runner 'type',
'run_as_root' and interpreter 'ext' all reject templates today, on this
branch and on v1.3.2, because each is consumed before rendering. These
tests keep the schema from being loosened past what the parser enforces.
main grew tt.uid/tt.gid after this branch was cut, so slice 3's allowed
tt.* list didn't know about them and the rebase left four tests failing:
the restriction rejected the canonical Docker user-mapping recipe
(args.build: UID={{ tt.uid }}). Both are host-global values resolved once
per run, exactly like tt.user_name, so they belong in the allowed set.

The Windows test's pinned wording also predates slice 1: runner fields
now render through Jinja's strict undefined, so it asserts the renderer's
message instead of the deleted regex path's.
Notes the decision-1 correction (no typed field is templatable, so the
schema stays narrow), what the audit actually found and fixed, the
prospective divergences part 3 will need entries for, and that the
reference worktree had to be recreated.
@kevinchannon
kevinchannon force-pushed the schema-validation-pipeline branch from d8f722e to c9ba142 Compare August 2, 2026 22:54
@claude

claude Bot commented Aug 2, 2026

Copy link
Copy Markdown

Review of #212 — Use JSON schema to validate incoming tasks

Reviewed via gh pr diff 212 against main (24 files, +4489/-937). Per the PR description and docs/plans/schema-validation-pipeline.md, this is explicitly a work-in-progress checkpoint (slices 0–5 + slice 6 part 1 of an 8-slice plan tracked in #43), opened early to see CI results rather than for merge. Reviewing it as such — mostly architectural soundness and risk-spotting rather than "is this done."

Overview

The PR replaces object-based recipe parsing (_parse_file/_parse_file_with_env building Task/Runner objects while recursing through imports) with a raw-dict merge phase (src/tasktree/raw_merge.py) that resolves imports, namespacing, run_in blankets, and variable-reference rewriting entirely on plain dicts before any object is constructed. Objects are then built from the merged tree in one pass. This is the necessary precondition for real JSON-schema validation (you can only run jsonschema.validate() against a dict, not a Task), and it's a big, well-telegraphed refactor — the tracking doc's slice-by-slice log is genuinely helpful for review.

New/notable pieces:

  • src/tasktree/raw_merge.py — the merge engine (590 lines, new).
  • src/tasktree/template_refs.py — a single generic {{ prefix.name }} reference walker (collect_template_refs) shared by variable-reachability, runner-field restriction, and (per the plan) future task hashing — replaces several bespoke enumerated-field-list scanners in parser.py.
  • Task-independent runner rendering (Executor._render_runner_field / task_config.build_runner_config) — runner/interpreter fields now render through the same Jinja path as task fields, restricted to env.*/tt.*.
  • Name-aware state pruning (state.py: TaskState.task_name) so pruning can distinguish "task deleted" from "task just wasn't part of this run" — this also fixes a real pre-existing v1.3.2 bug (state thrash on unrelated targeted runs) called out honestly in the expected-divergences list.
  • Task/runner/interpreter pruning to the reachable subtree on invocation, so parse-time defects in unreached parts of a recipe no longer block a run.
  • Schema fix: variables schema had integer and number as separate oneOf branches, so every int matched both and was rejected (e.g. port: 8080) — good catch, and it's backed by a 355-recipe fixture-corpus regression test (TestFixtureCorpus) that pins "the schema must never be stricter than the parser." That's a strong regression net for slice 6 parts 2–3.

Code quality / design

  • The class-based Runner hierarchy and polymorphic dispatch described in CLAUDE.md are respected — no new type-string branching introduced.
  • raw_merge.py is a clean, self-contained module with docstrings that explain why, not just what (e.g. why imported tasks merge before local, why namespacing walks values but not keys). Matches the repo's "comment on WHY" convention well.
  • template_refs.py is a good consolidation — one regex-based walker instead of N enumerated per-field scanners means new fields can't silently fall outside variable/runner-restriction scanning again.
  • Old dead code (_parse_file, _parse_file_with_env, ParsedFileResult, the five _rewrite_*_variable_references helpers, collect_reachable_tasks/collect_reachable_variables) is fully deleted rather than left behind — no half-migrated state.
  • Deliberate "over-match, never under-match" bias is stated explicitly and consistently applied across the walker, variable reachability, and pruning — a sound default for a scanner that feeds correctness-sensitive decisions (missing a ref breaks a run; an extra one is just wasted work).

Potential issues

  1. Silent drop of imported runners/variables/interpreters on a malformed local section (raw_merge.py, the three elif merged_*: if local_* is None: blocks). If a file's own runners:/variables:/interpreters: key is present but not a dict (and not None) — e.g. a typo like runners: docker instead of runners: {docker: {...}} — the merge neither raises nor keeps it, and any imported runners/variables/interpreters for that file are silently dropped rather than surfaced. Once part 3 wires jsonschema.validate() this becomes moot (schema will reject the malformed section), but until then it's a quiet failure mode worth a test either way (or a comment noting it's intentionally deferred to the schema).
  2. _apply_runner_transforms namespaces a string runner: value when present, and applies the blanket only when both runner and pin_runner are absent — worth double-checking the interaction with pin_runner: true + no runner at all: a pinned task with no runner should presumably still error (per the "pinned tasks must have runner specified" rule in CLAUDE.md). Confirm that validation still fires post-merge — didn't see it explicitly exercised in this diff; may already live elsewhere and be untouched, but worth a regression test given how much of the surrounding logic just moved.
  3. The reference-gate methodology (comparing against a pinned v1.3.2 worktree) is a good idea, but it lives entirely in a markdown doc with manual git worktree steps ("recreated at 36de66d... if it goes missing again"). That's a footgun for anyone other than the author reproducing verdicts later — worth eventually turning into a script or at least pinning the commit/tag in a constant rather than prose.
  4. The plan doc notes slice 6 parts 2–3 (merged-tree schema generator, wiring jsonschema.validate()) are not yet in this diff — so the schema itself still isn't enforced at runtime. Worth being explicit about that in the PR description (currently just says "premature... to allow assessment of CI") so reviewers don't go looking for the validation call.

Test coverage

Strong — roughly 190 new/changed test functions across unit (test_raw_merge.py with a parity-vs-object-path class, test_template_refs.py, test_runner_merge_cutover.py, test_task_merge_cutover.py, test_variable_reachability.py, test_runner_template_restriction.py, test_schema.py) and integration (test_state_pruning.py, test_unreachable_task_tolerance.py, test_builtin_variables.py). The fixture-corpus schema test and the "reference gate" divergence tracking are both good practices worth keeping as the plan continues — they turn "did I preserve v1.3.2 behavior except where documented" into an automatable check rather than a manual review burden.

Security

No concerns — this is parse-time/pure-dict logic, no new subprocess, network, or filesystem-write surface. _load_yaml correctly still uses yaml.safe_load.


Actions arising

  • Decide whether the silent-drop case for a malformed runners:/variables:/interpreters: section combined with imports (raw_merge.py, the elif merged_*: if local_* is None branches) needs a test/guard now, or is explicitly deferred to slice 6 part 3's schema wiring — add a one-line note either way.
  • Confirm "pinned task must specify runner" validation still runs correctly against the merged tree and add a regression test if it isn't already covered post-cutover.
  • Consider scripting the reference-gate worktree setup (~/repos/tasktree-ref at a pinned commit) instead of leaving it as manual steps in the plan doc, so future divergence checks are reproducible without tribal knowledge.
  • When slice 6 parts 2–3 land (merged-tree schema generator + wiring jsonschema.validate()), update the PR description/title to reflect that runtime enforcement is now actually active, since right now the schema file changes are unused by tt at runtime.
  • Continue the "expected divergences" tracking in the plan doc for the new prospective divergences already called out (non-string ports/volumes/env_vars, desc: 42, templated private/pin_runner/task_output, task_output not converted to TaskOutputTypes at parse) so they don't get lost before part 3 lands.

Imports merge into one tree before validation, so every name in it may
carry a namespace prefix ('build.compile') that the file schema forbids.
merged_tree_schema rewrites each name-keyed section's pattern to allow
dotted continuations, deriving the runtime schema from the authored one
so the two can't drift.

Unknown name patterns raise rather than passing through: a new
name-keyed section that the transform didn't namespace would silently
reject valid imported definitions.
Imports are consumed by the merge, so one surviving in the tree means the
merge left work undone. Dropping the property is enough to reject it --
the top level is additionalProperties: false -- and the anyOf branch that
allowed an imports-only file goes with it, since a merged tree can never
consist solely of imports.
Merges every fixture project that uses imports and validates the result,
so a namespacing rule the transform doesn't cover surfaces here rather
than when validation goes live at parse time. 28 projects merge cleanly
and all 28 validate; the rest are the merge's own error-handling
fixtures.

Checked the net bites: without the pattern rewrite, 28 of the 28 are
rejected.
The repo directory is itself called 'tasktree', so an __init__.py beside
it made the checkout root importable as the tasktree package -- and
pytest's rootdir insertion let it win over src/tasktree, so any test
importing tasktree from a package-rooted test directory got a package
with no modules in it. Nothing noticed because the e2e tests reach
tasktree through the CLI; a new test that imports it did.

The file held nothing but an empty docstring and was last touched by
docstring-sync churn. Full pyramid passes without it.
Runtime validation needs the schema file at hand, but it is authored at
the repo root -- outside the package -- because the READMEs point users'
editors at its raw-GitHub URL, which must keep resolving. The wheel now
force-includes it under the package, and the loader prefers that copy,
falling back to the authored one in a source checkout.
The force-include is invisible in a source checkout: the loader falls
back to the authored copy, so every test would pass while installed
users got a tasktree that cannot validate anything. This builds the
wheel and looks inside, including that the path matches the one the
loader looks in.
Notes what the generator does, the packaging decision and why it was
Kevin's to make, and the root __init__.py landmine cleared along the way.
Recipe validation is about to run inside parse_recipe, so jsonschema
stops being a test-only tool. The nested-invocation tests build
containers that pip-install tasktree's runtime deps by hand, so they need
it too -- without that, a tt invoked inside the container dies on import
while the outer run reports only a missing output file.
An empty recipe file is valid to tt, so the file schema's "at least one
section" rule -- editor guidance for someone starting a file -- must not
survive into the tree tt validates at parse time. Dropping the anyOf also
subsumes the imports branch removed with the imports property.
Both parse today but neither means what it looks like. A parameterized
dep's arguments must be a list or a mapping -- 'worker: msg=...' passes a
bare string, which parse_dependency_spec rejects the moment that dep is
invoked. And a runner's interpreter is 'interpreter:', so the runner
keyed 'shell:' was silently ignored, leaving the task on the default
interpreter rather than the bash the test names.

The parser has no key whitelist for either, so nothing complained; schema
validation does. Gate re-run for the dep change: unchanged verdict, 3
parity tests pass on v1.3.2 and the 2 intended divergences fail there.
jsonschema's own message dumps the failing subschema, which for a recipe
means pages of JSON -- a missing 'cmd' currently reports the entire task
schema. This keeps the part that identifies the problem and locates it in
the recipe's own terms: tasks.build.inputs[0], or tasks['build.release']
where the name itself contains dots.

Two constructs get reworded. A oneOf failure lists what the branches
accept ("Expected one of: Simple string value; ...") rather than printing
them; the enclosing description is deliberately not used, since for a
name-keyed section it describes the key, not the value. And 'not:
{required: [...]}', which the schema uses only to keep one runner kind's
fields off the others, becomes "'dockerfile' is not valid for this
runner's type".
These tests run tt inside a container against a mounted source tree, but
they mount only src/ -- and the recipe schema lives beside it at the repo
root, so a tt that validates recipes cannot find it there. Mounting the
schema dir alongside is what running from a source tree requires; an
installed wheel carries its own copy.
Names are not the schema's business: the merge reports a bad local name
against the file that defined it, and does so lazily, so a name nothing
references never breaks a run. Keeping the dotted-name patterns would
have overridden that -- an empty variable name, today a deferred
name error, would become a hard parse failure the moment validation
went live.

The rewritten patterns now accept any name and exist only to route each
value to its schema, except that 'default' must still fail to match,
since there it declares the default runner or interpreter rather than
naming one.
The schema stops being editor-only: parse_recipe now checks the merged
tree against it, so structural mistakes no hand-written check looks for
are caught. A misspelled task or runner field is the motivating case --
neither the parser nor the runner builders have a key whitelist, so
'outpts:' or a runner keyed 'shell:' was silently ignored and the task
quietly did the wrong thing.

Placement gives up nothing that already worked. It runs on the pruned
tree, so a defect in a task this invocation never reaches stays
tolerated; after the hand-written checks, so every message they produce
is unchanged and this slice stays additive; and before variables are
evaluated, so no 'eval:' command runs on the strength of a structurally
broken recipe.

The variable-form checks are the exception: they live inside variable
evaluation, so the schema now reaches those recipes first and nine
pinned tests move to the new wording. Reference gate: five new
rejections fail on v1.3.2 with "ValueError not raised", as intended.
Marks slice 6 done and records the placement correction (validation runs
after the hand-written checks, not before them, to stay additive), the
name-validity correction to decision 3, and the new expected-divergences
entries for what the schema now rejects.
@claude

claude Bot commented Aug 3, 2026

Copy link
Copy Markdown

Review summary

This PR (marked by the author as opened prematurely, purely to check CI) is a large, well-executed refactor that replaces the ad-hoc regex-based recipe parsing/substitution machinery with:

  1. A unified raw-dict import merge phase (raw_merge.py) that applies namespacing, run_in blanket overrides and pinned-runner rewriting as dict transforms before any Task/Runner object is built.
  2. A shared template reference walker (template_refs.py) used for pruning, hashing and the new runner variable-class restriction, so those three consumers can't disagree about what a subtree references.
  3. JSON Schema validation of the merged tree, derived programmatically from the authored file schema (recipe_schema.py) so the two schemas can't drift apart, with a message layer that turns jsonschema's raw errors into recipe-relative, human-readable text.
  4. Unifying runner/interpreter field substitution onto the Jinja renderer (_render_runner_field), deleting the old _substitute_builtin/_substitute_env regex helpers, plus a new parse-time check (check_runner_template_refs) that rejects per-task namespaces (arg.*, dep.*, self.*) in runner/interpreter definitions.
  5. Name-aware state pruning (TaskState.task_name) so a deleted task's stale cache entry can be distinguished from a defined-but-not-invoked task's entry.

Overall this is a real quality improvement: ~600 lines of duplicated/branchy parsing logic in parser.py are deleted and replaced by focused, well-documented, single-purpose modules. The docstrings consistently explain why (not just what), matching the project's Clean Code preference for named functions over comments. The 67 commits (per the linked plan doc, docs/plans/schema-validation-pipeline.md) show a genuinely incremental, test-as-you-go slice-by-slice history with a "reference gate" (diffing behaviour against v1.3.2) — exactly the workflow described in CLAUDE.md.

Strengths

  • DRY across consumers: collect_template_refs is shared by pruning, hashing, and the runner-restriction check, avoiding three independent regex implementations that could silently disagree.
  • Schema derivation instead of duplication: merged_tree_schema() transforms the authored file schema rather than hand-maintaining a second schema, with an explicit ValueError if a new name pattern isn't accounted for — a good guard against silent drift.
  • Error message quality: test_schema_errors.py specifically asserts messages stay short, name the recipe file/location in recipe terms, and never leak jsonschema internals (additionalProperties, should not be valid under..., etc.). This is exactly the kind of behavioural test the CLAUDE.md testing philosophy asks for.
  • Correct handling of tricky cases: diamond imports (non-circular), mutual/self circular imports, blanket run_in vs. pinned-runner precedence, and the new check_runner_template_refs restriction all have dedicated tests, including a "must not raise" acceptance test alongside the rejection tests.
  • Immutable-style recursion fix: _merge_file's import_stack = import_stack + [file_path] avoids the old _parse_file's mutate-then-.copy()-before-recursing pattern, which is a nice correctness simplification.

Potential issue: variable-reference namespacing misses non-trivial Jinja expressions

raw_merge._namespace_var_refs (built on VAR_REFERENCE_REWRITE_PATTERN) only rewrites a {{ var.X }} reference when it is the entire content of the template block:

VAR_REFERENCE_REWRITE_PATTERN = re.compile(r"(\{\{\s*var\.)([^\s}]+)(\s*}})")

Since the underlying renderer (rendering.py) is a full Jinja2 Environment, and template_refs.py's own extractor is explicitly designed to find refs anywhere inside a {{ ... }} block (its docstring/tests even cover {{ var.a if flag else var.b }}), a variable reference used with a filter or inside a larger expression in an imported file will not get namespaced during the merge, e.g.:

# lib.yaml, imported "as: lib"
variables:
  greeting: hello
tasks:
  hi:
    cmd: "echo {{ var.greeting | upper }}"   # not rewritten to var.lib.greeting

After merge, the cmd string still reads var.greeting (unnamespaced), while variables only carries the namespaced key lib.greeting. Because rendering.py uses StrictUndefined, this fails at render time with an "undefined" error if no root-level greeting variable exists — or, worse, silently resolves to the wrong variable if a same-named variable happens to exist at another scope, since nothing else catches it (it's not caught by _schema_validate, check_runner_template_refs doesn't apply to task fields, and _collect_referenced_variable_names/collect_template_refs — which does find the un-namespaced ref — would only either evaluate a nonexistent/wrong-scope variable or leave it unevaluated).

This exact whole-block-only limitation already existed pre-PR (the module's own comment notes it's "deliberately broader than the old per-field rewrite in parser.py", i.e. still narrower than the block-scanning approach template_refs.py demonstrates is achievable), so it isn't a regression this PR introduces — but the PR's own test suite (template_refs.py's tests, the plan doc's slice 2 notes) shows filtered/conditional Jinja expressions are an anticipated, tested pattern elsewhere in the same templating system, so the gap is more likely to be hit as this work lands. I couldn't find a test in test_raw_merge.py covering a var reference with a filter/expression across an import boundary, and no mention of this as an accepted divergence in docs/plans/schema-validation-pipeline.md.

Suggestion: either (a) make _namespace_var_refs scan for var.* references anywhere inside a {{ ... }} block (mirroring template_refs.py's block-then-reference two-step) so filters/expressions get rewritten too, or (b) if that's out of scope for this slice, add a test documenting the current limitation and consider raising a clear parse-time error when an imported file's var.* reference can't be safely namespaced (rather than failing silently or late at render time).

Minor notes

  • raw_merge._merge_file re-parses and reprocesses an imported file's whole subtree once per import site, so a "diamond" import (the same file imported from two different places) does double the YAML-load + transform work. Not a correctness issue (there's a test confirming diamond imports merge correctly), just a possible future perf/readability consideration if diamond imports become common in large recipes.
  • docs/plans/schema-validation-pipeline.md is a large, detailed planning/progress doc checked into the repo. Worth deciding whether it stays long-term in docs/ post-merge or gets trimmed once the migration is complete, so docs/ doesn't accumulate stale planning artifacts.
  • Since this PR is explicitly "opened prematurely" to check CI, it's worth confirming the full 67-commit history is intended to land as-is on main vs. being squash-merged — the CLAUDE.md incremental-commit philosophy seems well satisfied by the per-slice commit structure already, so a regular merge (preserving history) may be preferable to squashing all of this into one commit.

Test coverage

Coverage looks thorough: new unit tests for raw_merge (imports, namespacing, circular/diamond detection), template_refs, the runner-template restriction, schema error messages, and updated integration/e2e tests for nested invocations, recursion detection, state pruning, and unreachable-task tolerance. No unconditional test skips spotted, matching the project's testing philosophy.


Action items

  • Decide whether _namespace_var_refs/VAR_REFERENCE_REWRITE_PATTERN should be extended to rewrite var.* references embedded in larger Jinja expressions (filters, conditionals, concatenation) across import boundaries, not just whole-block references — and add a regression test for a filtered/expression var reference in an imported file.
  • If extending the rewrite is out of scope for this slice, document the current whole-block-only limitation explicitly (e.g. in the plan doc's divergences list) and consider a clear parse-time error instead of a silent/late failure when it's hit.
  • Confirm the intended merge strategy (regular merge preserving the 67-commit slice history vs. squash) once this PR is out of "premature" status.
  • Decide the long-term fate of docs/plans/schema-validation-pipeline.md (keep as historical record vs. trim once the migration lands).

The corpus test keyed each fixture by str(relative_to(...)), which is
backslash-separated on Windows, while the allowlist of intentionally
invalid fixtures is written with forward slashes -- so the one expected
rejection never matched its entry and both Windows jobs failed.

The added guard asserts every key is posix-style, which fails on Windows
under the old str() and passes under as_posix(); checked against
PureWindowsPath, since the two forms are identical on this machine.
Both tests execute the task they invoke, so hardcoding 'bash' made them a
test of whether bash is installed -- they have failed on Windows CI since
slice 5 landed, masked in the last two runs because the unit-test step
failed first. They now pick cmd.exe there, matching the pattern the
parser tests already use.

Gate re-run: verdict unchanged, both tests still pass on v1.3.2 while the
4 slice-5 tolerance divergences still fail there.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant