Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions openspec/changes/align-assertion-types/.openspec.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
schema: spec-driven
created: 2026-11-19
x-originating-issue: 7
skip_specs: true
50 changes: 50 additions & 0 deletions openspec/changes/align-assertion-types/proposal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Proposal

## Why

`test_mapping` classifies assertions into a vocabulary adapted from snake-eyes — `equality`, `comparison`, `membership`, `error_check`, plus `""` for 19 standard matchers it leaves unclassified (`toBeUndefined`, `toBeNull`, `toBeTruthy`, `toHaveLength`, `toHaveBeenCalled`, snapshots, …). That vocabulary is not Gaze's:

- Gaze's porting contract (`docs/porting/taxonomy-reference.md` § Assertion Types — a declared contract in `.uf/sibling-repos.yaml`) and the `assertion_type` enum of Gaze's JSON report schema (`internal/report/schema.go`) are exactly `equality`, `error_check`, `nil_check`, `diff_check`, `custom`. Reading Stone's `comparison`, `membership`, and `""` fall outside it.
- Gaze's external-analyzer adapter computes `AssertionDetectionConfidence` as the share of mapping rows with a non-empty `assertion_type` (`internal/adapter/quality.go` `classificationConfidence`). Every `""` row lowers it. On reading-stone's own suite, 324 of 3,820 rows (8.5%) were `""`; truthiness- or mock-heavy suites fare much worse.
- Gaze's Go-native detector never emits `""`: testify `Nil`/`NotNil` → `nil_check`, unknown forms → `custom`.

v0.2.0 is unpublished, so the wire vocabulary can change now without breaking any consumer.

## What Changes

- The closed kind set becomes Gaze's five values; `""`, `comparison`, and `membership` are removed.
- Routing principle: each vitest/jest/`node:assert` form takes the kind Gaze's Go detector assigns its testify analog (`classifyTestifyCall` + `mapKindToType`):
- `equality`: all value checks — previous `equality` forms plus the `toBeGreaterThan`/`toBeLessThan` families (testify `Greater`/`Less`), `toContain`/`toContainEqual`/`toMatch`/`assert.match`/`doesNotMatch` (`Contains`/`Regexp`), `toBeTruthy`/`toBeFalsy`/`assert(value)`/`assert.ok` (`True`/`False`), `toHaveLength` (`Len`), `toBeInstanceOf`/`toBeTypeOf` (`IsType`), `toHaveProperty`, `toBeNaN`.
- `error_check`: unchanged, plus `assert.ifError` (testify `NoError`).
- `nil_check`: `toBeNull`, `toBeUndefined`, `toBeDefined` (`Nil`/`NotNil`).
- `diff_check`: `toMatchSnapshot`, `toMatchInlineSnapshot` (go-cmp `Diff`).
- `custom`: mock occurrence/count forms (`toHaveBeenCalled[Times]`, `toHaveReturned[Times]`, `toBeCalled[Times]`), `toSatisfy`, `assert.fail`, and the miss path (unknown/custom matchers).
- Negation, `.resolves.`, and `.rejects.` semantics are unchanged.
- Amends the 004 normative sources in place: `spec.md` (US3 scenarios, FR-001, FR-008, FR-010, assumptions 1–2, 5), `data-model.md` (Sections 2–4, 6, 7.2, invariants), `contracts/test-mapping.md`, and an amendment note on `research.md` R6.

## Capabilities

### New Capabilities

None.

### Modified Capabilities

None in `openspec/specs/` — the `test_mapping` contract lives in `specs/004-test-mapping/`, amended in place (above).

## Impact

- `src/quality/types.ts` — `AssertionKind` union and the serializer's closed set.
- `src/quality/matchers.ts` — both normative tables and the miss path.
- `src/quality/assertions.ts`, `src/quality/inference.ts` — doc comments only (inference branches on `error_check` alone; `assert.ifError` rows now take the error fallback chain, as an error-channel observation should).
- `tests/quality/*` — kind expectations; the doc-parity suite re-diffs the code against the amended data-model tables.
- Wire: `test_mapping` `assertion_type` values change; `analyze`, `classify_signals`, and every other method are unaffected. Gaze consumes `assertion_type` as an opaque string, so no Gaze change is needed.
- Cross-repo: snake-eyes carries the same divergence (`comparison`, `membership`, `identity`, `generic`); tracked as a follow-up there.

## Constitution Alignment

- **I. Protocol Fidelity — PASS**: aligns the wire vocabulary with Gaze's own taxonomy and report schema; deterministic, table-driven.
- **II. Detection Accuracy — PASS**: detection is unchanged (same assertions, same rows); classification now feeds Gaze's metric correctly instead of under-reporting recognized assertions.
- **III. TypeScript-Native Analysis — N/A**: no compiler-API changes.
- **IV. Testability — PASS**: coverage strategy — the existing doc-parity suite diffs `matchers.ts` against the amended data-model tables row by row; assertion, inference, and pipeline suites pin per-kind fixtures including the new `nil_check`, `diff_check`, `custom`, and `ifError → error_check` rows; 85% gate unchanged.
- **V. Analysis Safety — PASS**: no new dependencies; no I/O changes.
10 changes: 10 additions & 0 deletions openspec/changes/align-assertion-types/tasks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Tasks

## 1. Implementation

- [x] 1.1 `src/quality/types.ts`: `AssertionKind` = `equality | error_check | nil_check | diff_check | custom`; serializer closed set updated; `src/quality/matchers.ts`: both tables re-routed per data-model Section 4 and the miss path → `custom`; doc comments in `assertions.ts`/`inference.ts` updated
- [x] 1.2 Update `tests/quality/*` expectations (doc parity, assertions, inference, pipeline, types) and add pins for `nil_check` (`toBeUndefined`), `diff_check` (`toMatchSnapshot`), `custom` (unknown matcher, `toHaveBeenCalled`), and `assert.ifError` → `error_check`

## 2. Gates

- [x] 2.1 Run the CI parity gates — `npx biome check src/ tests/`, `npm run build`, `npx vitest run --coverage` — all pass at ≥85%; verify on reading-stone itself that no `test_mapping` row carries an `assertion_type` outside Gaze's five values
2 changes: 1 addition & 1 deletion specs/004-test-mapping/contracts/test-mapping.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ advertised in the same change:
| `test_function` | Describe-qualified title: enclosing `describe` titles joined `" > "`, then the test title; bare title at top level; `""` for non-static titles |
| `test_file` | Root-relative POSIX path of the test file |
| `assertion_location` | `"<relPath>:<line>"` of the assertion's anchor — the outermost wrapper of the matcher/assert invocation (`await` and parenthesized wrappers unwrapped upward) — no column |
| `assertion_type` | `equality` \| `comparison` \| `membership` \| `error_check` \| `""` per the pinned matcher table ([data-model.md](data-model.md) Section 4) |
| `assertion_type` | `equality` \| `error_check` \| `nil_check` \| `diff_check` \| `custom` (Gaze's assertion-type set; never empty) per the pinned matcher table ([data-model.md](data-model.md) Section 4) |
| `target_function` / `target_package` | The paired production function's declared name and `derivePackage` dotted module path — the same identity `analyze` reports |
| `side_effect_type` | The inferred canonical taxonomy name (provenance passes then the fallback chains — [data-model.md](data-model.md) Section 7); always present, never an alias |
| `confidence` | The pairing strategy's fixed integer: naming-exact `90`, naming-case-insensitive `70`, direct-calls `80`, import-graph `75` |
Expand Down
127 changes: 71 additions & 56 deletions specs/004-test-mapping/data-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ keeps TypeScript identifiers identical to the wire names, the
| `test_function` | `string` | The describe-qualified title (Section 5.1); possibly empty (non-static titles) |
| `test_file` | `string` | Root-relative POSIX path of the test file |
| `assertion_location` | `string` | `"<relPath>:<line>"` — 1-based line of the assertion's anchor (the outermost wrapper of the matcher/assert invocation, `await` spellings included); no column component |
| `assertion_type` | `string` | Exactly one of `equality`, `comparison`, `membership`, `error_check`, `""` — never `generic`, never `null` |
| `assertion_type` | `string` | Exactly one of `equality`, `error_check`, `nil_check`, `diff_check`, `custom` — Gaze's assertion-type set; never empty, never `null` |
| `target_function` | `string` | The target record's declared name |
| `target_package` | `string` | `derivePackage(targetRecord.relPath)` — the dotted module path |
| `side_effect_type` | `string` | One of the 48 canonical taxonomy names (aliases never emitted) |
Expand All @@ -78,7 +78,7 @@ export interface MappingRow {
readonly test_function: string;
readonly test_file: string;
readonly assertion_location: string;
readonly assertion_type: AssertionKind; // "" | "equality" | "comparison" | "membership" | "error_check"
readonly assertion_type: AssertionKind; // "equality" | "error_check" | "nil_check" | "diff_check" | "custom"
readonly target_function: string;
readonly target_package: string;
readonly side_effect_type: SideEffectType;
Expand All @@ -93,8 +93,7 @@ export interface TestMappingResult {

**Serializer** (`serializeMappingRow`, `src/quality/types.ts`): the
`serializeSignal` house pattern — fail loud (Constitution I):
`assertion_type` membership in the closed five-value kind set
(including `""`); `side_effect_type` canonical membership through
`assertion_type` membership in the closed five-value kind set; `side_effect_type` canonical membership through
`resolveSideEffectType` (a canonical name passes, anything else throws
— aliases are construction-time bugs that must not reach the wire);
`confidence` an integer in [0, 100]; every string field type-checked.
Expand All @@ -104,26 +103,42 @@ An empty analysis is a `{"mappings": []}` success (US1-8).

## 3. Assertion-type vocabulary

The closed kind set (FR-001/FR-008, spec assumptions 1–2):
The closed kind set (FR-001/FR-008, spec assumptions 1–2) is Gaze's
assertion-type set, verbatim from the porting contract
(`docs/porting/taxonomy-reference.md` § Assertion Types) and the
`assertion_type` enum of Gaze's JSON report schema:

| Kind | Denotation |
|---|---|
| `equality` | The assertion pins an exact value: `toBe` (documented strict equality / `Object.is`), structural equality, closeness, object-shape matching, with-argument mock forms |
| `comparison` | Ordered relations only: strictly the `toBeGreaterThan` / `toBeLessThan` families |
| `membership` | Containment and pattern matching: `toContain` family, `toMatch` |
| `error_check` | The assertion observes the error channel: `toThrow` family, every matcher under `.rejects`, `assert.throws` family |
| `""` | Recognized assertion, unclassified kind: property, truthiness, type, bare-occurrence, count, and snapshot forms; unknown or custom matchers |

Two deliberate divergences from the reference analyzer's six-value set:
no `generic` (the empty string is the unclassified convention — the
platform's assertion-detection metric counts rows carrying a recognized
type, so `""` feeds it exactly), and no `identity` (Python's `is`
bucket has no distinct vitest/jest analog — `toBe` is documented
strict equality and routes to `equality`).
| `equality` | The assertion checks a value: exact and structural equality, closeness, object shape, with-argument mock forms, ordered relations, containment and pattern matching, truthiness, type and instance checks, length and property checks |
| `error_check` | The assertion observes the error channel: `toThrow` family, every matcher under `.rejects`, `assert.throws` family, `assert.ifError` |
| `nil_check` | Null/undefined presence: `toBeNull`, `toBeUndefined`, `toBeDefined` |
| `diff_check` | Structural diff against a recorded expectation: `toMatchSnapshot`, `toMatchInlineSnapshot` |
| `custom` | Framework-specific or unrecognized: mock-interaction occurrence and count forms, predicate matchers, `assert.fail`, unknown or custom matchers |

**Routing principle** (amended 2026-11-19, OpenSpec change
`align-assertion-types`): each vitest/jest/`node:assert` form routes to
the kind Gaze's Go-native detector assigns its testify analog
(`internal/quality/assertion.go` `classifyTestifyCall` +
`internal/quality/mapping.go` `mapKindToType`): `Equal`, `Contains`,
`Regexp`, `Len`, `Empty`, `True`/`False`, `Greater`/`Less`, `IsType`,
`Implements`, `Zero`, `InDelta` → `equality`; `NoError`/`Error*` →
`error_check`; `Nil`/`NotNil` → `nil_check`; `cmp.Diff` → `diff_check`;
anything unrecognized → `custom`. A TypeScript suite and an equivalent
Go suite therefore score identically on Gaze's assertion-detection
confidence, which counts rows with a non-empty `assertion_type`
(`internal/adapter/quality.go` `classificationConfidence`).

Superseded (pre-amendment): the four-kind set `equality`,
`comparison`, `membership`, `error_check` plus `""` for unclassified,
adapted from the reference analyzer's six-value set. It was not
Gaze's vocabulary: `comparison` and `membership` fall outside Gaze's
report-schema enum, and every `""` row lowered Gaze's
assertion-detection confidence for 19 standard matchers.

Negation semantics (assumption 3): `.not.` and `not`-prefixed assert
functions preserve the underlying kind — the `comparison` kind denotes
ordered relations only, never inequality. `resolves`/`rejects`
functions preserve the underlying kind — negation flips polarity,
never the kind. `resolves`/`rejects`
(assumption 4): `.resolves.` preserves the matcher's kind; `.rejects.`
routes to `error_check` (the assertion is about the rejection channel).

Expand All @@ -134,7 +149,7 @@ routes to `error_check` (the assertion is about the rejection channel).
FR-007 pins this table as normative data. `src/quality/matchers.ts`
transcribes it verbatim; `tests/quality/matchers.test.ts` diffs the
module against this section. The lookup is total: a form absent from
the table routes to `""` through the miss path — never an exception,
the table routes to `custom` through the miss path — never an exception,
never a drop (US3-6).

### 4.1 `expect()` matchers
Expand All @@ -155,36 +170,36 @@ never a drop (US3-6).
| `toBeCalledWith` | `equality` (legacy jest alias) |
| `lastCalledWith` | `equality` (legacy jest alias) |
| `nthCalledWith` | `equality` (legacy jest alias) |
| `toBeGreaterThan` | `comparison` |
| `toBeGreaterThanOrEqual` | `comparison` |
| `toBeLessThan` | `comparison` |
| `toBeLessThanOrEqual` | `comparison` |
| `toContain` | `membership` |
| `toContainEqual` | `membership` |
| `toMatch` | `membership` |
| `toBeGreaterThan` | `equality` |
| `toBeGreaterThanOrEqual` | `equality` |
| `toBeLessThan` | `equality` |
| `toBeLessThanOrEqual` | `equality` |
| `toContain` | `equality` |
| `toContainEqual` | `equality` |
| `toMatch` | `equality` |
| `toThrow` | `error_check` |
| `toThrowError` | `error_check` |
| `toThrowErrorMatchingSnapshot` | `error_check` |
| `toThrowErrorMatchingInlineSnapshot` | `error_check` |
| `toBeTruthy` | `""` |
| `toBeFalsy` | `""` |
| `toBeNull` | `""` |
| `toBeUndefined` | `""` |
| `toBeDefined` | `""` |
| `toBeNaN` | `""` |
| `toBeInstanceOf` | `""` |
| `toHaveBeenCalled` | `""` |
| `toHaveBeenCalledTimes` | `""` |
| `toHaveReturned` | `""` |
| `toHaveReturnedTimes` | `""` |
| `toBeCalled` | `""` (legacy jest alias) |
| `toBeCalledTimes` | `""` (legacy jest alias) |
| `toHaveProperty` | `""` |
| `toHaveLength` | `""` |
| `toMatchSnapshot` | `""` |
| `toMatchInlineSnapshot` | `""` |
| `toBeTypeOf` | `""` (vitest) |
| `toSatisfy` | `""` (vitest) |
| `toBeTruthy` | `equality` |
| `toBeFalsy` | `equality` |
| `toBeNull` | `nil_check` |
| `toBeUndefined` | `nil_check` |
| `toBeDefined` | `nil_check` |
| `toBeNaN` | `equality` |
| `toBeInstanceOf` | `equality` |
| `toHaveBeenCalled` | `custom` |
| `toHaveBeenCalledTimes` | `custom` |
| `toHaveReturned` | `custom` |
| `toHaveReturnedTimes` | `custom` |
| `toBeCalled` | `custom` (legacy jest alias) |
| `toBeCalledTimes` | `custom` (legacy jest alias) |
| `toHaveProperty` | `equality` |
| `toHaveLength` | `equality` |
| `toMatchSnapshot` | `diff_check` |
| `toMatchInlineSnapshot` | `diff_check` |
| `toBeTypeOf` | `equality` (vitest) |
| `toSatisfy` | `custom` (vitest) |

**Modifier semantics** (applied to the invoked matcher's kind):

Expand All @@ -208,22 +223,22 @@ routing — the syntactic form routes identically.

| Assert form | Kind |
|---|---|
| `assert(value)` (bare) | `""` |
| `assert.ok` | `""` |
| `assert.ifError` | `""` |
| `assert.fail` | `""` |
| `assert(value)` (bare) | `equality` |
| `assert.ok` | `equality` |
| `assert.ifError` | `error_check` |
| `assert.fail` | `custom` |
| `assert.equal` / `assert.notEqual` | `equality` |
| `assert.strictEqual` / `assert.notStrictEqual` | `equality` |
| `assert.deepEqual` / `assert.notDeepEqual` | `equality` |
| `assert.deepStrictEqual` / `assert.notDeepStrictEqual` | `equality` |
| `assert.match` / `assert.doesNotMatch` | `membership` |
| `assert.match` / `assert.doesNotMatch` | `equality` |
| `assert.throws` | `error_check` |
| `assert.doesNotThrow` | `error_check` |
| `assert.rejects` | `error_check` |
| `assert.doesNotReject` | `error_check` |

`not`-prefixed functions preserve their base kind (US3-7). An
unknown member call on an assert binding routes to `""`. The asserted
unknown member call on an assert binding routes to `custom`. The asserted
expression — provenance's subject — is the FIRST argument of every
assert form.

Expand Down Expand Up @@ -268,7 +283,7 @@ discipline, never a request crash.

| Field | Type | Notes |
|---|---|---|
| `kind` | `AssertionKind` | The classified kind or `""` (Section 4) |
| `kind` | `AssertionKind` | The classified kind (Section 4) |
| `line` | `number` | 1-based line of the assertion's anchor — the outermost wrapper of the matcher/assert invocation (`await` and parenthesized wrappers unwrapped upward), so every matcher invocation on one shared chain reports the SAME anchor (US1-1; the row's `assertion_location` line, no column on the wire) |
| `character` | `number` | 0-based character of that same outermost anchor — the row-order tie-break |
| `assertedExpression` | `Node \| null` | The provenance subject: `expect(...)`'s first argument, or the assert call's first argument; `null` when absent |
Expand Down Expand Up @@ -395,7 +410,7 @@ By assertion type alone — never provenance:
ErrorReturn if present, else ErrorSignal if present, else ErrorReturn
```

### 7.2 Value rows (`equality`, `comparison`, `membership`, `""`)
### 7.2 Value rows (`equality`, `nil_check`, `diff_check`, `custom`)

**Standard fallback chain** (FR-010 — the reference's final
semantics, generator-aware from the start):
Expand Down Expand Up @@ -508,8 +523,8 @@ responses including a full second pipeline run.
one pair shares the pair's strategy confidence; there is no
per-assertion confidence.
6. **Kind totality** (FR-007/FR-008): every expect matcher invocation
and recognized assert call classifies to a kind or `""` — the
table's miss path is total.
and recognized assert call classifies to a kind — the
table's miss path (`custom`) is total.
7. **Static analysis only** (FR-012, Constitution V): test files are
parsed by the compiler API inside the bounded hermetic host, never
executed or imported — pinned by canary fixtures.
Expand Down
7 changes: 7 additions & 0 deletions specs/004-test-mapping/research.md
Original file line number Diff line number Diff line change
Expand Up @@ -427,6 +427,13 @@ is the actual operand in every standard form.

## R6: The matcher table — normative data, totality, and the unknown-form fallback

> **Amended 2026-11-19** (OpenSpec change `align-assertion-types`): the
> kind set is now Gaze's `equality` / `error_check` / `nil_check` /
> `diff_check` / `custom`, routed by parity with Gaze's Go-native testify
> routing, and the miss path routes to `custom` instead of `""`. The
> decision record below is kept as written; data-model.md Sections 3–4
> are the current normative source.

**Decision**: `matchers.ts` is pure fixed data — the FR-007 routing
table: every standard vitest/jest matcher and `node:assert` function
routed to a classification kind or explicitly to none (`""`),
Expand Down
Loading