Use JSON schema to validate incoming tasks - #212
Conversation
ReviewThis PR is explicitly a WIP checkpoint (slices 0–3 of the schema-validation-pipeline plan, issue #43), so I focused the review on the code that's actually landed rather than on the plan's not-yet-built slices. Summary of the change
Code quality / best practices
Potential bugs / issues
Performance
Security
Test coverage
Action items
|
- collect_template_refs keeps numeric continuation segments, so
positional references like {{ self.inputs.0 }} retain their index
(mattered for slice 5/7 reuse, not the current prefix check).
- The double check_runner_template_refs walk (whole runner config in
build_recipe_runner, interpreter subtree again in
_parse_inline_interpreter) is documented as intentional.
- test_builtin_vars_in_runner_volumes pins os.getlogin and asserts the
exact substituted USER_NAME_VAR value. Reference gate: passes on
v1.3.2 (parity).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Review action items addressed in d2b44e4:
Full pyramid run after the changes: unit+integration 1647 passed / 4 conditional skips; e2e 71 passed / 5 conditional skips. |
Review of PR #212 — "Use JSON schema to validate incoming tasks" (slices 0-3, WIP)Thanks for the clear per-slice commit history and the self-documenting plan doc - it made this easy to follow even without the originating conversation. This is explicitly an in-progress WIP PR per the description, so I focused on the substance of slices 1-3 that has landed rather than the overall roadmap. Summary of what's in this PR
Correctness issue: the "runners can't see per-task values" guarantee doesn't actually hold everywhereThe core safety property this slice is building towards is that runner/interpreter fields must not depend on per-task state, because a runner is shared across tasks. Two gaps undermine that today:
Suggest: for (1), either call Smaller notes
Performance / securityNo concerns - Action items
|
Both verifications pass: the Jinja renderer covers every namespace the unified path needs, and no fixture or test uses arg./dep./self. templates in runner or interpreter fields. Slice 1's field list is expanded to match the regex path's true coverage (dockerfile, context, build/run args and interpreter cmd/preamble, in addition to the originally listed fields). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Volumes now go through rendering.render with a runner-scoped context (env + tt only, via the new build_runner_config) instead of the chained regex substitutions. Per-task namespaces (arg/dep/self) in a volume are now an error instead of being silently left in the mount string. Reference gate: parity test passes on v1.3.2; the arg-rejection test fails there as expected (documented in the plan's divergences list). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same move as volumes: env_vars values go through the runner-scoped Jinja context (env + tt), so per-task namespaces in an env_vars value now error instead of silently reaching the container as literal text. Reference gate: parity test passes on v1.3.2; arg-rejection test fails there as expected (covered by the documented divergence). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same move as volumes and env_vars. Reference gate: parity passes on v1.3.2; arg-rejection fails there as expected (documented divergence). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same move as volumes/env_vars/ports. Reference gate: parity passes on v1.3.2; arg-rejection fails there as expected (documented divergence). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The shared subst() closure now uses the runner-scoped Jinja render, migrating docker build/run args, interpreter cmd/preamble, dockerfile and context in one move. Pre-existing parity tests for these fields pass unchanged; new rejection tests pin the arg.* error behaviour. Reference gate: parity via the existing (unchanged) tests; both new rejection tests fail on v1.3.2 as expected (documented divergence). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_substitute_builtin and _substitute_env have no callers now that all runner fields render through the Jinja engine. The underlying substitution.py functions remain (still used on the parse-time path). No behaviour change, so no reference-gate run (no behaviour-level test added or changed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Slice 2 of the schema validation pipeline (issue #43): the generic reference walker, starting with regex extraction from a single string. Purely additive - nothing calls it yet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dict keys are walked too, per the over-matching bias: a template in a key position (e.g. env_vars) must not be silently missed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Chases referenced variables' definitions to a fixpoint so consumers see everything a subtree transitively depends on. Undefined names are kept but not chased - discovery is not validation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The generic walker already sees templates inside eval commands, read
paths and env defaults; the { env: NAME } form needs a special case
because it names its env var as a bare string. Cyclic definitions
terminate via the chased set.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Slice 3 of the schema validation pipeline (decision 4): runners and interpreters are shared across tasks and render once, so arg/dep/self and per-task tt builtins may not appear in their definitions. The check uses the slice-2 walker; nothing calls it yet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
build_recipe_runner now runs check_runner_template_refs on the raw config, covering section runners and inline task runners. The builtin_vars_runner_volumes fixture drops its tt.task_name env var (now-forbidden per-task builtin) in favour of tt.user_name. Reference gate (v1.3.2): the four rejection tests fail there with 'ValueError not raised' - the recipes were silently accepted - matching the expected-divergences entries; the per-task tt divergence is added to the list. The acceptance test and the updated volumes test pass on the reference. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_parse_inline_interpreter runs check_runner_template_refs, covering the interpreters section, task-level inline interpreters and runner interpreter fields. Error wording generalised to name interpreters. Reference gate (v1.3.2): all six rejection tests fail there with 'ValueError not raised' (recipes silently accepted), matching the expected-divergences entries; both acceptance tests pass there. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Status header updated (slices 0-3 done), the full-pyramid test bar and per-increment commit permission recorded in process decisions, gate practicalities from slices 2-3 added to the reference-arbiter section, and slice 4 gets the parser entry points surveyed this session. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- collect_template_refs keeps numeric continuation segments, so
positional references like {{ self.inputs.0 }} retain their index
(mattered for slice 5/7 reuse, not the current prefix check).
- The double check_runner_template_refs walk (whole runner config in
build_recipe_runner, interpreter subtree again in
_parse_inline_interpreter) is documented as intentional.
- test_builtin_vars_in_runner_volumes pins os.getlogin and asserts the
exact substituted USER_NAME_VAR value. Reference gate: passes on
v1.3.2 (parity).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CircularImportError moves to raw_merge (its natural home once the merge phase owns imports); parser re-exports it so existing imports keep working. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same rules as _parse_file: undotted deps get the namespace prefix; dotted deps are prefixed only when their root segment is one of the importing file's own import namespaces, otherwise kept as absolute references. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mirrors _parse_file: an imported file's runners come along only when a pinned task references them; imported 'default' declarations are dropped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Var refs are rewritten by a generic tree walk over each imported file's local content (values only), deliberately broader than the old per-field rewrite: dep argument templates and inline definitions are covered too. VAR_REFERENCE_REWRITE_PATTERN moves to raw_merge; parser imports it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
parse_recipe gains prune_unreachable (opt-in from execute_dynamic_task only): when the root task exists, unreachable tasks are dropped from the merged tree before construction, so their defects are tolerated. --list/--show/--tree leave the flag off and still validate the whole file (--show/--tree pass a root task for lazy variable evaluation, so pruning cannot key off root_task alone). defined_task_names now captures the pre-pruning universe for name-aware state pruning. Gate: both tolerance tests fail on v1.3.2 as intended (it validated every task); the three validation-coverage parity tests pass there. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
prune_unreferenced_runners/_interpreters (raw_merge) run right after
task pruning: the 'default' declarations and their targets always
survive, task/runner {use:} references are chased from survivors only,
and the CLI --runner/--interpreter override names are threaded through
parse_recipe as keep-hints so overrides keep working. Listing/showing
paths still build and validate every definition.
Gate: the four tolerance tests fail on v1.3.2 as intended; the six
override/coverage parity tests pass there.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
evaluate_variables now derives the reachable set with collect_reachable_task_names on the merged tree and discovers referenced variables with collect_template_refs over the reachable tasks' subtrees plus their referenced runners (and the default runner). The object-based collect_reachable_tasks / collect_reachable_variables and their enumerated field list are deleted. Gate: parity (unreachable tasks' variables stay unevaluated) passes on v1.3.2; the coverage divergence test fails there with 'Variable not defined' - the old enumerated list only looked inside Docker runner definitions, so lazy parsing broke on vars referenced only by host runner fields. Divergence recorded in the plan. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Also repurposes TestParityWithObjectPath as a merged-tree consistency check (its cutover parity role is obsolete now parse_recipe builds on the merge). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The variables value schema listed 'integer' and 'number' as separate
oneOf branches, so every integer matched both and failed validation --
'variables: {port: 8080}' was rejected outright (4 fixtures in this repo
hit it). 'number' covers integers, so the redundant branch goes.
Found by auditing the file schema against what the parser accepts, ahead
of wiring the schema in at runtime.
A runner definition carried a 'default: true' key. The parser has no
whitelist of runner keys, so it was silently ignored -- the default
runner is declared as 'runners: {default: <name>}' at section level, not
inside a definition. The schema (additionalProperties: false) rightly
rejects it, so it would break once validation runs at parse time.
355 recipes under tests/fixtures are exactly the recipes tt is expected to handle, so they are the sharpest available check that the file schema is not stricter than the parser -- the property slice 6 needs before the schema starts rejecting recipes at runtime. Rejections are compared against an explicit allowlist of fixtures the schema is meant to reject, so a new deliberate negative forces a conscious entry rather than quietly widening the tolerated set. Checked the net bites: artificially narrowing the variables schema to strings turns the expected 1 rejection into 26.
Plan decision 1 assumed users legitimately write templates in typed
fields, so the schema would have to accept 'native type OR {{...}}'.
Probing the parser says otherwise: min/max, arg 'type', runner 'type',
'run_as_root' and interpreter 'ext' all reject templates today, on this
branch and on v1.3.2, because each is consumed before rendering. These
tests keep the schema from being loosened past what the parser enforces.
main grew tt.uid/tt.gid after this branch was cut, so slice 3's allowed
tt.* list didn't know about them and the rebase left four tests failing:
the restriction rejected the canonical Docker user-mapping recipe
(args.build: UID={{ tt.uid }}). Both are host-global values resolved once
per run, exactly like tt.user_name, so they belong in the allowed set.
The Windows test's pinned wording also predates slice 1: runner fields
now render through Jinja's strict undefined, so it asserts the renderer's
message instead of the deleted regex path's.
Notes the decision-1 correction (no typed field is templatable, so the schema stays narrow), what the audit actually found and fixed, the prospective divergences part 3 will need entries for, and that the reference worktree had to be recreated.
d8f722e to
c9ba142
Compare
Review of #212 — Use JSON schema to validate incoming tasksReviewed via OverviewThe PR replaces object-based recipe parsing ( New/notable pieces:
Code quality / design
Potential issues
Test coverageStrong — roughly 190 new/changed test functions across unit ( SecurityNo concerns — this is parse-time/pure-dict logic, no new subprocess, network, or filesystem-write surface. Actions arising
|
Imports merge into one tree before validation, so every name in it may
carry a namespace prefix ('build.compile') that the file schema forbids.
merged_tree_schema rewrites each name-keyed section's pattern to allow
dotted continuations, deriving the runtime schema from the authored one
so the two can't drift.
Unknown name patterns raise rather than passing through: a new
name-keyed section that the transform didn't namespace would silently
reject valid imported definitions.
Imports are consumed by the merge, so one surviving in the tree means the merge left work undone. Dropping the property is enough to reject it -- the top level is additionalProperties: false -- and the anyOf branch that allowed an imports-only file goes with it, since a merged tree can never consist solely of imports.
Merges every fixture project that uses imports and validates the result, so a namespacing rule the transform doesn't cover surfaces here rather than when validation goes live at parse time. 28 projects merge cleanly and all 28 validate; the rest are the merge's own error-handling fixtures. Checked the net bites: without the pattern rewrite, 28 of the 28 are rejected.
The repo directory is itself called 'tasktree', so an __init__.py beside it made the checkout root importable as the tasktree package -- and pytest's rootdir insertion let it win over src/tasktree, so any test importing tasktree from a package-rooted test directory got a package with no modules in it. Nothing noticed because the e2e tests reach tasktree through the CLI; a new test that imports it did. The file held nothing but an empty docstring and was last touched by docstring-sync churn. Full pyramid passes without it.
Runtime validation needs the schema file at hand, but it is authored at the repo root -- outside the package -- because the READMEs point users' editors at its raw-GitHub URL, which must keep resolving. The wheel now force-includes it under the package, and the loader prefers that copy, falling back to the authored one in a source checkout.
The force-include is invisible in a source checkout: the loader falls back to the authored copy, so every test would pass while installed users got a tasktree that cannot validate anything. This builds the wheel and looks inside, including that the path matches the one the loader looks in.
Notes what the generator does, the packaging decision and why it was Kevin's to make, and the root __init__.py landmine cleared along the way.
Recipe validation is about to run inside parse_recipe, so jsonschema stops being a test-only tool. The nested-invocation tests build containers that pip-install tasktree's runtime deps by hand, so they need it too -- without that, a tt invoked inside the container dies on import while the outer run reports only a missing output file.
An empty recipe file is valid to tt, so the file schema's "at least one section" rule -- editor guidance for someone starting a file -- must not survive into the tree tt validates at parse time. Dropping the anyOf also subsumes the imports branch removed with the imports property.
Both parse today but neither means what it looks like. A parameterized dep's arguments must be a list or a mapping -- 'worker: msg=...' passes a bare string, which parse_dependency_spec rejects the moment that dep is invoked. And a runner's interpreter is 'interpreter:', so the runner keyed 'shell:' was silently ignored, leaving the task on the default interpreter rather than the bash the test names. The parser has no key whitelist for either, so nothing complained; schema validation does. Gate re-run for the dep change: unchanged verdict, 3 parity tests pass on v1.3.2 and the 2 intended divergences fail there.
jsonschema's own message dumps the failing subschema, which for a recipe
means pages of JSON -- a missing 'cmd' currently reports the entire task
schema. This keeps the part that identifies the problem and locates it in
the recipe's own terms: tasks.build.inputs[0], or tasks['build.release']
where the name itself contains dots.
Two constructs get reworded. A oneOf failure lists what the branches
accept ("Expected one of: Simple string value; ...") rather than printing
them; the enclosing description is deliberately not used, since for a
name-keyed section it describes the key, not the value. And 'not:
{required: [...]}', which the schema uses only to keep one runner kind's
fields off the others, becomes "'dockerfile' is not valid for this
runner's type".
These tests run tt inside a container against a mounted source tree, but they mount only src/ -- and the recipe schema lives beside it at the repo root, so a tt that validates recipes cannot find it there. Mounting the schema dir alongside is what running from a source tree requires; an installed wheel carries its own copy.
Names are not the schema's business: the merge reports a bad local name against the file that defined it, and does so lazily, so a name nothing references never breaks a run. Keeping the dotted-name patterns would have overridden that -- an empty variable name, today a deferred name error, would become a hard parse failure the moment validation went live. The rewritten patterns now accept any name and exist only to route each value to its schema, except that 'default' must still fail to match, since there it declares the default runner or interpreter rather than naming one.
The schema stops being editor-only: parse_recipe now checks the merged tree against it, so structural mistakes no hand-written check looks for are caught. A misspelled task or runner field is the motivating case -- neither the parser nor the runner builders have a key whitelist, so 'outpts:' or a runner keyed 'shell:' was silently ignored and the task quietly did the wrong thing. Placement gives up nothing that already worked. It runs on the pruned tree, so a defect in a task this invocation never reaches stays tolerated; after the hand-written checks, so every message they produce is unchanged and this slice stays additive; and before variables are evaluated, so no 'eval:' command runs on the strength of a structurally broken recipe. The variable-form checks are the exception: they live inside variable evaluation, so the schema now reaches those recipes first and nine pinned tests move to the new wording. Reference gate: five new rejections fail on v1.3.2 with "ValueError not raised", as intended.
Marks slice 6 done and records the placement correction (validation runs after the hand-written checks, not before them, to stay additive), the name-validity correction to decision 3, and the new expected-divergences entries for what the schema now rejects.
Review summaryThis PR (marked by the author as opened prematurely, purely to check CI) is a large, well-executed refactor that replaces the ad-hoc regex-based recipe parsing/substitution machinery with:
Overall this is a real quality improvement: ~600 lines of duplicated/branchy parsing logic in Strengths
Potential issue: variable-reference namespacing misses non-trivial Jinja expressions
VAR_REFERENCE_REWRITE_PATTERN = re.compile(r"(\{\{\s*var\.)([^\s}]+)(\s*}})")Since the underlying renderer ( # lib.yaml, imported "as: lib"
variables:
greeting: hello
tasks:
hi:
cmd: "echo {{ var.greeting | upper }}" # not rewritten to var.lib.greetingAfter merge, the This exact whole-block-only limitation already existed pre-PR (the module's own comment notes it's "deliberately broader than the old per-field rewrite in parser.py", i.e. still narrower than the block-scanning approach Suggestion: either (a) make Minor notes
Test coverageCoverage looks thorough: new unit tests for Action items
|
The corpus test keyed each fixture by str(relative_to(...)), which is backslash-separated on Windows, while the allowlist of intentionally invalid fixtures is written with forward slashes -- so the one expected rejection never matched its entry and both Windows jobs failed. The added guard asserts every key is posix-style, which fails on Windows under the old str() and passes under as_posix(); checked against PureWindowsPath, since the two forms are identical on this machine.
Both tests execute the task they invoke, so hardcoding 'bash' made them a test of whether bash is installed -- they have failed on Windows CI since slice 5 landed, masked in the last two runs because the unit-test step failed first. They now pick cmd.exe there, matching the pattern the parser tests already use. Gate re-run: verdict unchanged, both tests still pass on v1.3.2 while the 4 slice-5 tolerance divergences still fail there.
This PR is being opened prematurely to allow assessment of the state of CI tests against this work.
Eventually thë whole feature will be present and we can merge it.