Skip to content

docs(testing): hurl is scaffolding; target-native suites are the goal#52

Merged
koolamusic merged 1 commit into
mainfrom
chore/testing-strategy
May 9, 2026
Merged

docs(testing): hurl is scaffolding; target-native suites are the goal#52
koolamusic merged 1 commit into
mainfrom
chore/testing-strategy

Conversation

@koolamusic

Copy link
Copy Markdown
Contributor

Summary

Captures the strategic note that hurl is not a long-term
requirement for candy. Hurl was the right v0.1 conformance gate
(cross-target proof, no per-language re-authoring), but it isn't
the test surface a team adopting candy will actually run in their
day-to-day loop.

Candy should generate the test suite in the target's idiomatic
test framework alongside the feature implementation.

Two test tiers per target

Tier Focus Idiomatic surface (per target)
1 (focus, integration) HTTP-level end-to-end Go: `testing` + `httptest`. Rust: `#[tokio::test]` + axum-test. TS: vitest + `app.fetch(req)`. Python: pytest + httpx / FastAPI `TestClient`.
2 (supporting, unit) Per-policy, per-flow, per-actor Already proven on the Go (`policies_test.go`) and Rust (`#[cfg(test)] mod tests`) auth targets.

Each hurl scenario transliterates mechanically to one tier-1 test.
The codegen has the spec; it knows the routes, status mappings,
and expected body shapes — it can emit the target-native suite
from the same inputs.

Hurl's new role

Not deleted. Demoted to the cross-target reference contract.
A single hurl run still verifies a generated backend matches the
conformance contract. Teams running on a single target use the
language-native suite for daily work; hurl is the shared fence-line
for multi-target conformance.

Spec evolutions that would enrich tier-1 generation

Out of scope for this PR; flagged in the doc:

  • `controller X { examples: ... }` — HTTP-shape examples direct
    on controllers (today examples live on `policy` blocks).
  • `flow X { properties: ... }` — property-based testing surface.
  • `mock external X { Op returns ... }` — declarative external
    stubbing for tier-1 tests.
  • `seed:` blocks at feature level — declarative test-mode seed
    data.

Out of scope

No code changes. No commitment to specific test framework choices
beyond the conventional defaults. The doc is a strategic framing;
concrete codegen prompts for tier-1 emission land in a follow-up.

Why this matters

The diff between "candy emits hurl I have to learn" and "candy
emits `_test.go` I already read every day" is the difference
between a niche tool and a default.

Combined with brownfield mode (PR #51), this gives candy a
realistic adoption story: applies to existing code, ships in your
team's existing test idiom.

Refs

  • PR docs(cli): document green and brown codegen modes #51 — green/brown CLI modes (testing strategy is what makes
    brown adoption palatable).
  • `evals/README.md` — current hurl-based conformance contract.
  • `.retrospective/phase-alpha-codegen.md` §8 — "no CI for codegen
    targets yet"; tier-1 native suites are the natural CI surface.

Captures the strategic note: hurl was the right v0.1 conformance
gate (cross-target proof with no per-language re-authoring), but it
isn't the long-term contract a team adopting candy will run in
their day-to-day loop. Candy should generate the test suite in the
target's idiomatic framework alongside the feature implementation.

Establishes two test tiers per target:

- Tier 1 (focus, integration): HTTP-level, language-native — Go
  testing + httptest, Rust #[tokio::test] + axum-test, vitest +
  hono.fetch, pytest + httpx. Each hurl scenario transliterates
  mechanically to one tier-1 test.

- Tier 2 (supporting, unit): per-policy, per-flow, per-actor.
  Already proven on the Go and Rust auth targets — every spec
  policy example becomes a unit test.

Hurl is demoted to its proper role: the cross-target reference
contract. Teams running on a single target use only the
language-native suite.

Names spec-side evolutions that would enrich tier-1 generation
beyond simple transliteration: `controller examples`,
`flow properties`, `mock external X`, declarative `seed:` blocks.
Marks all of these as not in immediate scope.

The "diff between hurl and `_test.go`" framing is the adoption
metric — a tool teams already-read becomes the default; a tool
they have to learn stays niche. Same logic as the green/brown
distinction in cli-modes.md, applied to the test surface.

No code changes — strategy doc only.
@koolamusic koolamusic merged commit 1c02976 into main May 9, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant