diff --git a/CHANGELOG.md b/CHANGELOG.md index 4919d0127..e740ed15b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,26 @@ version number before tagging the release. ## [current] +### Added +- `std.mutation` — Tier-1 (text-based) mutation-testing driver, adopted + from the sunsetting aeocha repo's `contrib/mutate` (the last real + technology blocking that repo's archival). One entry point, + `mutation.run(sut, test, lib_dir)`, plus a runnable front-end at + `examples/mutation-testing/mutate.ae` and a worked calc example. + Perturbs the SUT source one padded-operator (or string-literal) site + at a time, rebuilds, re-runs the suite, and classifies each mutant + killed / survived / no-compile via std.spec's structured report + (`AE_SPEC_FORMAT`/`AE_SPEC_REPORT`), so a non-compiling mutant never + masquerades as a kill; the SUT is restored byte-identical. Sub-builds + honour `AE_BIN` (in-tree harnesses point it at `build/ae`). The two + deterministic aeocha regression fixtures came along + (`tests/integration/mutation_testing/`): the operator fixture pins + `1/2 … 50%` with a MUL→DIV survivor + md5-identical restore, the + strings fixture pins the string-boundary skip (an operator inside a + string literal is never mutated as code). docs/mutation-testing.md + documents usage, the operator set, and the honest Tier-1 limits; the + AST-level upgrade path is the reason the tool now lives in-tree. + ### Fixed - FreeBSD: the `ae` driver, `aetherc` module resolver, and `ae help` now locate the running executable via the `KERN_PROC_PATHNAME` sysctl diff --git a/docs/mutation-testing.md b/docs/mutation-testing.md new file mode 100644 index 000000000..063e9971d --- /dev/null +++ b/docs/mutation-testing.md @@ -0,0 +1,198 @@ +# Mutation testing — `std.mutation` + +Mutation testing measures **how good your tests are**. It changes the code +under test (the "SUT") one tiny edit at a time — `+` becomes `-`, `>` becomes +`<` — and re-runs your test suite for each change. A change your tests *catch* +is a **killed** mutant (good). A change that slips through is a **survivor** — +proof your tests have a gap. + +`std.mutation` is a small library (one entry point, `mutation.run`) with a +runnable front-end at `examples/mutation-testing/mutate.ae`. It was adopted +from the sunsetting aeocha repo's `contrib/mutate` and uses `std.spec` purely +as an **oracle**, through the structured-report contract (`AE_SPEC_FORMAT` / +`AE_SPEC_REPORT`, docs/testing.md). + +## Why it's an outer driver (not a test helper) + +The compiler has already run by the time any test executes, so a mutant can't +be produced in-process — there's no source or AST left to perturb, only +compiled machine code. Mutation therefore has to, once per mutant: + +1. edit the SUT **source on disk** and clear the build cache, +2. **`ae check`** the test — if the mutant doesn't type-check, it's *no-compile* + (excluded from the score; a mutant that won't build was never really tested), +3. **`ae build`** the test to a binary and run it with + `AE_SPEC_FORMAT=aeocha` + `AE_SPEC_REPORT=` set, **reading the + structured report** from that file rather than scraping stdout. The report's + `failed=N` is the verdict: `N > 0` → killed, `N == 0` → survived. + +`mutation.run` is that outer loop; your test file is completely unaware +mutation is happening. Using the structured report (not just a process exit +code) is what lets the tool tell *killed* (a test actually failed) apart from +*no-compile* (the mutation produced invalid code) — so a non-compiling mutant +never inflates the score by masquerading as a kill. + +> Compiler note: the no-compile gate greps `ae check`'s output for an `error[` +> diagnostic rather than trusting its exit code. On the current `ae`, both +> `ae check` and `ae build` can print a compile error to stderr yet still exit 0 +> (and `build` will even emit a binary linked against a stale module) — see +> issue #953. Grepping the diagnostic is the reliable signal until that's fixed. + +## Usage + +```bash +ae run examples/mutation-testing/mutate.ae -- [lib_dir] +``` + +`lib_dir` is a module search dir handed to the per-mutant sub-builds via +`AETHER_LIB_DIR`; it defaults to the SUT's directory (which is usually right — +the test's `import ` resolves there). The `ae` used for the +sub-builds is `ae` on PATH, overridable with the `AE_BIN` environment variable +(the in-tree regression harness points it at `build/ae`). + +Or from a program / build script: + +``` +import std.mutation + +survivors = mutation.run(sut_path, test_path, "") +// survivors >= 0: run completed (0 = every compiling mutant killed) +// survivors < 0: aborted (unreadable SUT, or baseline suite fails) +``` + +Worked example (from the repo root): + +```bash +ae run examples/mutation-testing/mutate.ae -- \ + examples/mutation-testing/lib/calc.ae \ + examples/mutation-testing/lib/calc_test.ae +``` + +`examples/mutation-testing/lib/` holds `calc.ae` (the code under test) plus +`calc_test.ae` (its `std.spec` suite). Mutation testing asks how well that +suite tests that code. Output: + +``` +Aether mutation testing (std.mutation) + SUT: examples/mutation-testing/lib/calc.ae + test: examples/mutation-testing/lib/calc_test.ae + + baseline: suite passes on unmutated SUT ✓ + + killed calc.ae:8 ADD->SUB + killed calc.ae:11 SUB->ADD + killed calc.ae:11 LT->GT + + 3/3 mutants killed — mutation score 100% +``` + +100% — every single-operator change to `calc.ae` was caught by `calc_test.ae`. +Each line is one mutant, anchored to its source location: `ADD->SUB` flips the +`+` in `add`; `SUB->ADD` flips the `0 - n` in `abs`; `LT->GT` flips the +`n < 0` guard. The suite asserts both branches of `abs` and an addition with a +negative, so all three are killed. + +### What a survivor looks like + +A survivor is the interesting case — it's a *test gap*. Delete the +negative-input case from `calc_test.ae` (the `abs(-7)` assertion), and the +`SUB->ADD` mutant suddenly survives: + +``` + killed calc.ae:8 ADD->SUB + SURVIVED calc.ae:11 SUB->ADD + killed calc.ae:11 LT->GT + + 2/3 mutants killed — mutation score 66% + 1 survived (test gaps): + - calc.ae:11 SUB->ADD +``` + +`SUB->ADD` flips the `0 - n` in `abs` to `0 + n` — which only changes the +result for a *negative* input. With `abs(-7)` removed, nothing feeds `abs` a +negative, so the mutant goes unnoticed. (Note `LT->GT` is still killed: +flipping the `n < 0` guard sends the positive `abs(7)` down the negation +branch, and that case is still tested.) Add the negative-input assertion back +and `SUB->ADD` returns to killed. That is the whole point — survivors are your +missing tests. + +## Mutation operators (core set) + +Operator mutators are matched **whitespace-padded** (`" + "`, `" > "`), so +normal Aether spacing is required for a site to be seen — and a padded +operator that sits *inside a string literal* is skipped (the tool tracks +string boundaries, so `"a + b"` in your code is never mutated as arithmetic). + +| Operator | Mutation | +|---|---| +| `+` ↔ `-` | arithmetic | +| `*` → `/` | arithmetic | +| `>` `<` `>=` `<=` | comparison flips | +| `==` ↔ `!=` | equality flips | +| `&&` ↔ `\|\|` | boolean | + +String-literal mutators target the literal's *content* (quotes preserved): + +| Mutator | Mutation | +|---|---| +| `STR->EMPTY` | a non-empty `"foo"` → `""` (catches tests that don't pin the returned string) | +| `EMPTY->NONEMPTY` | an empty `""` → a sentinel (catches an unchecked empty-string case) | + +## Reading the result + +- **Mutation score** = killed / (killed + survived). Higher is better; 100% + means every single-operator change that *compiled* was caught by some test. +- **Survivors** are your to-do list: each one is reported as + `file:line MUTATION` so you can jump straight to the unguarded code. Either + add a test that distinguishes it, or convince yourself it's an *equivalent + mutant* (see caveats). +- **No-compile** mutants (the mutation produced invalid code) are reported as + `(N excluded — did not compile)` and left out of the denominator — they were + never really tested, so they neither help nor hurt the score. With the core + operator set they're rare (most operator swaps stay valid), but the category + keeps the score honest when they happen. + +## Honest limitations + +This is a Tier-1, text-based tool. Know what it does and doesn't do: + +- **Text, not AST.** Operators are matched as padded tokens, so `++`, `+=`, + and operators that abut other characters are skipped. The tool *is* + string-boundary aware — a padded operator inside a `"..."` literal won't be + mutated as code, and string-literal mutators only touch real literals. The + remaining blind spot is **comments**: a `"..."` or a padded operator written + in a `//` comment is still treated as source, so it can produce a harmless + false mutant (it changes nothing the suite sees → survives). Keep + operator/quote characters out of comments in a SUT you mutate, or expect a + stray survivor. +- **Equivalent mutants.** Some changes don't alter behaviour (e.g. `<` → `<=` + on a boundary your code never reaches). They "survive" without being real + gaps. This is inherent to mutation testing, not a bug here. +- **Slow.** Every mutant pays a cache-clear + `ae check` + `ae build` + run — + and the cache-clear is mandatory (an imported-module edit doesn't invalidate + the cache, so you'd otherwise test a stale build). Three compiler invocations + per mutant, no warm-cache reuse — point it at a focused SUT, not your whole + codebase. +- **Crash safety.** The driver restores the original SUT at the end of the run + (the regression fixtures verify byte-identical restore). But it mutates the + real file in place, so if the driver is killed mid-run (Ctrl-C, OOM), the + SUT is left mutated — recover with `git checkout `. Run it on a + clean working tree. +- **POSIX-only.** The oracle shells out through `/bin/sh` and + `os.run_pipe_drain_and_wait`; Windows is not supported. + +## The upgrade path (why this lives in the aether tree) + +Every limitation above is a compiler-adjacency problem: text-based mutation +can produce mutants a parser would never emit; every mutant pays full +rebuilds; there's no equivalent-mutant detection; and the #953 grep-workaround +exists only because the tool used to sit outside the compiler. Precise, +false-mutant-free mutation needs **AST-level edits** — operator sites from the +real parser, positions from `ae inspect`-grade data, perhaps an `ae mutate` +subcommand. Adopting the Tier-1 tool into the tree puts that upgrade path +where the compiler is. + +Regression coverage: `tests/integration/mutation_testing/` (a deterministic +operator fixture pinning `1/2 … 50%` with a MUL→DIV survivor and byte-identical +SUT restore, plus a string-literal fixture covering the boundary-awareness +skip). diff --git a/examples/mutation-testing/lib/calc.ae b/examples/mutation-testing/lib/calc.ae new file mode 100644 index 000000000..c1968e87f --- /dev/null +++ b/examples/mutation-testing/lib/calc.ae @@ -0,0 +1,12 @@ +// Code under test for the mutation-testing demo (docs/mutation-testing.md). +// The point of mutation testing is to check how well calc_test.ae tests +// this code. NB: comments in a mutated SUT deliberately avoid padded +// operator tokens and quote characters, so the text-based mutator sees +// only the real code sites below. +exports(add, abs) + +add(a: int, b: int) -> int { return a + b } + +abs(n: int) -> int { + if n < 0 { return 0 - n } else { return n } +} diff --git a/examples/mutation-testing/lib/calc_test.ae b/examples/mutation-testing/lib/calc_test.ae new file mode 100644 index 000000000..884e509ab --- /dev/null +++ b/examples/mutation-testing/lib/calc_test.ae @@ -0,0 +1,29 @@ +// The std.spec suite for calc.ae. The mutation driver edits calc.ae on +// disk and re-runs THIS unchanged file once per mutant; its structured +// report (AE_SPEC_FORMAT/AE_SPEC_REPORT) is the oracle. +import std.spec +import calc + +main() { + fw = spec.init() + + spec.describe(fw, "calc.add") { + spec.it("adds two positives") callback { + spec.assert_eq(calc.add(2, 3), 5, "2 plus 3") + } + spec.it("adds with a negative") callback { + spec.assert_eq(calc.add(10, -4), 6, "10 plus minus 4") + } + } + + spec.describe(fw, "calc.abs") { + spec.it("leaves positives unchanged") callback { + spec.assert_eq(calc.abs(7), 7, "abs of 7") + } + spec.it("flips negatives") callback { + spec.assert_eq(calc.abs(-7), 7, "abs of minus 7") + } + } + + spec.run_summary(fw) +} diff --git a/examples/mutation-testing/mutate.ae b/examples/mutation-testing/mutate.ae new file mode 100644 index 000000000..39fb77afe --- /dev/null +++ b/examples/mutation-testing/mutate.ae @@ -0,0 +1,39 @@ +// mutate — runnable front-end for std.mutation, the Tier-1 (text-based) +// mutation-testing driver. See docs/mutation-testing.md for the full +// story (what mutation testing is, the operator set, honest limits). +// +// Usage: +// ae run examples/mutation-testing/mutate.ae -- [lib_dir] +// +// lib_dir is a module search dir handed to the per-mutant sub-builds via +// AETHER_LIB_DIR; it defaults to the SUT's directory. Exit codes: +// 0 — run completed (survivors, if any, are listed in the output) +// 1 — aborted (SUT unreadable, or the unmutated suite fails/no-compiles) +// 2 — usage error +// +// Worked example (the same calc as docs/mutation-testing.md): +// ae run examples/mutation-testing/mutate.ae -- \ +// examples/mutation-testing/lib/calc.ae \ +// examples/mutation-testing/lib/calc_test.ae + +import std.mutation +import std.os + +main() { + // os.aether_args_get(0) is the binary path; user args start at 1. + ac = os.aether_args_count() + if ac < 3 { + println("usage: ae run examples/mutation-testing/mutate.ae -- [lib_dir]") + os.exit(2) + } + sut_path = os.aether_args_get(1) + test_path = os.aether_args_get(2) + incl = "" + if ac >= 4 { + incl = os.aether_args_get(3) + } + rc = mutation.run(sut_path, test_path, incl) + if rc < 0 { + os.exit(1) + } +} diff --git a/std/mutation/module.ae b/std/mutation/module.ae new file mode 100644 index 000000000..605cda12c --- /dev/null +++ b/std/mutation/module.ae @@ -0,0 +1,596 @@ +// std.mutation — a Tier-1 (text-based) mutation-testing driver for +// std.spec-tested code. Adopted from aeocha's contrib/mutate as that +// repo sunsets (github.com/aether-lang-dev/aeocha). +// +// Mutation testing measures how good your TESTS are: it perturbs the +// code under test (the "SUT") one change at a time and checks whether +// the test suite NOTICES. A mutant the tests catch is "killed"; one +// that slips through "survived" — a survivor marks a gap in the tests. +// +// This is an OUTER driver, deliberately. The compiler has already run +// by the time any test executes, so mutation can't happen in-process — +// it must edit the SUT *source on disk*, rebuild, and re-run the suite, +// once per mutant. std.spec is the oracle, via its STRUCTURED report +// (not just an exit code): the driver builds the test to a binary, runs +// it with AE_SPEC_FORMAT/AE_SPEC_REPORT set (the documented env-file +// transport, docs/testing.md), and reads `failed=N` from the `version=1` +// report. That lets it classify three outcomes — killed (a test +// failed), survived (suite passed, so a test gap), and no-compile (the +// mutation produced invalid code, excluded from the score) — so a +// non-compiling mutant never masquerades as a kill. See `_oracle`. +// +// Entry point: run(sut_path, test_path, lib_dir) — see the runnable +// front-end at examples/mutation-testing/mutate.ae: +// ae run examples/mutation-testing/mutate.ae -- [lib_dir] +// (lib_dir "" defaults to the SUT's directory.) The `ae` used for the +// per-mutant sub-builds is `ae` on PATH, overridable via the AE_BIN +// environment variable (an in-tree harness points it at build/ae). +// +// Mutators: code-operator swaps (+/- , */ , compare flips, ==/!=, &&/||) +// matched whitespace-padded, plus string-literal mutators (non-empty -> +// "", "" -> sentinel). The tool is string-boundary aware — a padded +// operator inside a "..." literal is NOT mutated as code, and the string +// mutators touch only real literals. +// +// HONEST LIMITATIONS (Tier-1, text-based; docs/mutation-testing.md): +// - Comments are still treated as source: a "..." or padded operator +// written in a // comment can yield a harmless false mutant (changes +// nothing the suite sees -> survives). Keep them out of a SUT. +// - `++`, `+=`, operators abutting other chars: skipped (need padding). +// - No equivalent-mutant detection: a mutation that doesn't change +// behaviour (e.g. `<` -> `<=` on a boundary never hit) "survives" +// without being a real gap. Universal mutation-testing noise. +// - Each mutant pays a cache-clear + check + build + run. No warm +// cache (the stale-cache rule forces the clear), so it's SLOW on +// big suites. Start with a focused SUT. +// The upgrade path out of Tier-1 is AST-level mutation — operator sites +// from the real parser instead of padded-token scans — which is why the +// driver now lives in the aether tree, next to the compiler. + +exports(run) + +import std.string +import std.fs +import std.list +import std.os + +// Oracle classification codes. +// SURVIVED — the mutant compiled and the suite still PASSED (a gap) +// KILLED — the mutant compiled and the suite FAILED (good) +// NOCOMPILE — the mutant did not type-check (excluded from the score) +const SURVIVED = 0 +const KILLED = 1 +const NOCOMPILE = 2 + +// The `ae` driver used for per-mutant sub-builds: AE_BIN if set (an +// in-tree test harness points it at build/ae), else `ae` from PATH. +_ae_bin() -> string { + b = os.getenv("AE_BIN") + if b == null { return "ae" } + if string.equals(b, "") == 1 { return "ae" } + return b +} + +// _oracle(test_path, incl, bin) classifies one already-written mutant by +// running the std.spec test against it. `incl` is a module search dir +// passed as AETHER_LIB_DIR to the sub-`ae check`/`ae build` (must hold +// any non-std modules the test imports, the SUT itself included). +// Three steps: +// +// 1. ae check — the no-compile gate. We gate on `check`, not `build`, +// because `ae build` currently accepts an entry program whose +// imported module fails to compile and links a stale/valid version +// (#953). `check` honestly reports the import error. +// 2. ae build the test to a binary. +// 3. run it with AE_SPEC_FORMAT=aeocha + AE_SPEC_REPORT= set — +// std.spec's run_summary writes the `version=1` report to that +// file (the documented env-file transport, docs/testing.md). We +// parse `failed=N` from the header: N>0 => KILLED, N==0 => +// SURVIVED. This uses the structured report rather than scraping +// stdout or trusting only an exit code. +// +// Cache is cleared first every time — an imported-module edit does not +// invalidate ~/.aether/cache, so without this every mutant would be +// tested against the previously compiled SUT. +_oracle(test_path: string, incl: string, bin: string) -> int { + os.system("rm -rf ~/.aether/cache") + ae = _ae_bin() + + // (1) no-compile gate. We gate on the PRESENCE OF `error[` IN OUTPUT, + // not the exit code: on this compiler both `ae check` and `ae build` + // return exit 0 even when an imported module fails to compile — they + // print the diagnostic to stderr but don't fail (#953). So we grep + // stderr for an `error[` diagnostic; grep -q exits 0 when it finds + // one, which means the mutant does NOT compile. + chk = "AETHER_LIB_DIR='${incl}' '${ae}' check '${test_path}' 2>&1 | grep -q 'error\\['" + if os.system(chk) == 0 { + return NOCOMPILE + } + + // (2) build to a fresh binary (exit code is unreliable too; we rely + // on the check gate above and on whether a runnable binary appears). + os.system("rm -f '${bin}'") + bld = "AETHER_LIB_DIR='${incl}' '${ae}' build '${test_path}' -o '${bin}' >/dev/null 2>&1" + os.system(bld) + if os.system("test -x '${bin}'") != 0 { + return NOCOMPILE + } + + // (3) run + read the structured report. We spawn the binary + // through `/bin/sh -c "... exec >/dev/null 2>&1"` so the + // child's own ✓/✗ test output is silenced; the AE_SPEC_* env pair + // (assignments before `exec` apply to the exec'd program) makes + // run_summary write the report to a sibling file the redirect + // can't touch. A mutant that crashes before run_summary leaves no + // file — fs.read yields "" and _report_failed returns -1, which + // classifies as SURVIVED, exactly as an empty report would. + rpt = "${bin}.report" + os.system("rm -f '${rpt}'") + argv = list.new() + list.add(argv, "-c") + list.add(argv, "AE_SPEC_FORMAT=aeocha AE_SPEC_REPORT='${rpt}' exec '${bin}' >/dev/null 2>&1") + _pipe, _rc, drain_err = os.run_pipe_drain_and_wait("/bin/sh", argv, null) + if string.equals(drain_err, "") == 0 { + return NOCOMPILE + } + report, _rerr = fs.read(rpt) + os.system("rm -f '${rpt}'") + failed = _report_failed(report) + if failed > 0 { return KILLED } + return SURVIVED +} + +// Parse `failed=N` from a version=1 report header. Returns -1 if absent +// (treated by callers as "no usable report"). +_report_failed(report: string) -> int { + key = "failed=" + pos = string.index_of_from(report, key, 0) + if pos < 0 { return -1 } + start = pos + string.length(key) + rlen = string.length(report) + // read digits until newline + end = start + scanning = 1 + while scanning == 1 { + if end >= rlen { + scanning = 0 + } else { + c = string.char_at_n(report, rlen, end) + if c >= 48 && c <= 57 { + end = end + 1 + } else { + scanning = 0 + } + } + } + if end == start { return -1 } + return string.get_int(string.substring(report, start, end)) +} + +// --------------------------------------------------------------------------- +// String helpers — mutate exactly the Nth CODE occurrence of a token, +// where "code" means not inside a "..." string literal. +// --------------------------------------------------------------------------- + +// Is byte offset `off` inside a "..." string literal? Walk from 0 +// tracking quote state; inside a string a backslash escapes the next +// byte (so \" does not close the string). 34 = '"', 92 = '\'. +// This is what stops an operator (` + `) that happens to sit inside a +// string from being mutated as if it were code — a false mutant. +_in_string(src: string, off: int) -> int { + slen = string.length(src) + i = 0 + instr = 0 + while i < off { + c = string.char_at_n(src, slen, i) + if instr == 1 { + if c == 92 { + i = i + 2 + } else { + if c == 34 { instr = 0 } + i = i + 1 + } + } else { + if c == 34 { instr = 1 } + i = i + 1 + } + } + return instr +} + +// 1-based line number containing byte offset `off` (count newlines +// before it). Used to anchor each mutant to a source location. +_line_at(src: string, off: int) -> int { + slen = string.length(src) + line = 1 + i = 0 + while i < off { + if i < slen { + if string.char_at_n(src, slen, i) == 10 { line = line + 1 } + } + i = i + 1 + } + return line +} + +// Byte offset of the n-th CODE occurrence of `find` (string literals +// skipped), or -1. Mirrors _replace_nth's scan so the runner can report +// the mutation's line without re-deriving it from the mutated text. +_offset_of_nth(src: string, find: string, n: int) -> int { + flen = string.length(find) + k = 0 + pos = string.index_of_from(src, find, 0) + while pos >= 0 { + if _in_string(src, pos) == 0 { + if k == n { return pos } + k = k + 1 + } + pos = string.index_of_from(src, find, pos + flen) + } + return -1 +} + +// Count occurrences of `find` that are NOT inside a string literal. +_count(src: string, find: string) -> int { + flen = string.length(find) + n = 0 + pos = string.index_of_from(src, find, 0) + while pos >= 0 { + if _in_string(src, pos) == 0 { n = n + 1 } + pos = string.index_of_from(src, find, pos + flen) + } + return n +} + +// Replace the n-th (0-based) CODE occurrence of `find` with `repl` +// (occurrences inside string literals don't count toward n and are +// never touched). Returns "" if there is no n-th code occurrence. +_replace_nth(src: string, find: string, repl: string, n: int) -> string { + slen = string.length(src) + flen = string.length(find) + k = 0 + found = -1 + pos = string.index_of_from(src, find, 0) + while pos >= 0 { + if _in_string(src, pos) == 0 { + if k == n { + found = pos + pos = -1 + } else { + k = k + 1 + pos = string.index_of_from(src, find, pos + flen) + } + } else { + pos = string.index_of_from(src, find, pos + flen) + } + } + if found < 0 { return "" } + head = string.substring(src, 0, found) + tail = string.substring(src, found + flen, slen) + return "${head}${repl}${tail}" +} + +// --------------------------------------------------------------------------- +// String-literal mutation. Two operators that catch tests which never +// pin down a returned/used string: +// STR->EMPTY — a non-empty "foo" becomes "" +// EMPTY->NONEMPTY — an empty "" becomes a sentinel +// Implemented on the same escape-aware scan as _in_string. We mutate +// the literal's CONTENT, keeping the surrounding quotes. +// --------------------------------------------------------------------------- + +// Count string literals whose emptiness matches want_empty (1 = only +// "", 0 = only non-empty). +_count_strings(src: string, want_empty: int) -> int { + slen = string.length(src) + i = 0 + n = 0 + while i < slen { + c = string.char_at_n(src, slen, i) + if c == 34 { + j = i + 1 + scanning = 1 + while scanning == 1 { + if j >= slen { scanning = 0 } + else { + cj = string.char_at_n(src, slen, j) + if cj == 92 { j = j + 2 } + else { if cj == 34 { scanning = 0 } else { j = j + 1 } } + } + } + empty = 0 + if j - i - 1 == 0 { empty = 1 } + if empty == want_empty { n = n + 1 } + i = j + 1 + } else { + i = i + 1 + } + } + return n +} + +// Byte offset (opening quote) of the n-th string literal matching +// want_empty, or -1. Mirrors _mutate_nth_string's scan for line reporting. +_offset_of_nth_string(src: string, n: int, want_empty: int) -> int { + slen = string.length(src) + i = 0 + k = 0 + while i < slen { + c = string.char_at_n(src, slen, i) + if c == 34 { + j = i + 1 + scanning = 1 + while scanning == 1 { + if j >= slen { scanning = 0 } + else { + cj = string.char_at_n(src, slen, j) + if cj == 92 { j = j + 2 } + else { if cj == 34 { scanning = 0 } else { j = j + 1 } } + } + } + empty = 0 + if j - i - 1 == 0 { empty = 1 } + if empty == want_empty { + if k == n { return i } + k = k + 1 + } + i = j + 1 + } else { + i = i + 1 + } + } + return -1 +} + +// Replace the CONTENT of the n-th (0-based) string literal matching +// want_empty with `content` (quotes preserved). "" if no n-th match. +_mutate_nth_string(src: string, n: int, want_empty: int, content: string) -> string { + slen = string.length(src) + i = 0 + k = 0 + while i < slen { + c = string.char_at_n(src, slen, i) + if c == 34 { + j = i + 1 + scanning = 1 + while scanning == 1 { + if j >= slen { scanning = 0 } + else { + cj = string.char_at_n(src, slen, j) + if cj == 92 { j = j + 2 } + else { if cj == 34 { scanning = 0 } else { j = j + 1 } } + } + } + empty = 0 + if j - i - 1 == 0 { empty = 1 } + if empty == want_empty { + if k == n { + // [i] and [j] are the opening/closing quotes; splice + // new content between them. + head = string.substring(src, 0, i + 1) + tail = string.substring(src, j, slen) + return "${head}${content}${tail}" + } + k = k + 1 + } + i = j + 1 + } else { + i = i + 1 + } + } + return "" +} + +// --------------------------------------------------------------------------- +// Mutation operator table. Each entry is a (find, repl) pair of +// whitespace-padded tokens. Order matters for multi-char operators: +// we mutate ">=" / "<=" / "==" / "!=" as their own padded tokens so a +// ">" rule never half-eats a ">=". +// --------------------------------------------------------------------------- + +// Final path component of `path` (the SUT's file name), for anchoring +// mutants as `:` rather than a bare occurrence index. +_basename(path: string) -> string { + plen = string.length(path) + last = -1 + p = string.index_of_from(path, "/", 0) + while p >= 0 { + last = p + p = string.index_of_from(path, "/", p + 1) + } + if last < 0 { return path } + return string.substring(path, last + 1, plen) +} + +// Classify one written mutant and report/tally it. `loc` is the source +// anchor (`file.ae:line`); `label` is the mutation (`ADD->SUB`). +_classify(loc: string, label: string, test_path: string, incl: string, bin: string, + killed: ptr, survived: ptr, nocompile: ptr, survivors: ptr) { + verdict = _oracle(test_path, incl, bin) + if verdict == SURVIVED { + ref_set(survived, ref_get(survived) + 1) + list.add(survivors, "${loc} ${label}") + println(" SURVIVED ${loc} ${label}") + } else { + if verdict == KILLED { + ref_set(killed, ref_get(killed) + 1) + println(" killed ${loc} ${label}") + } else { + // NOCOMPILE — the mutation produced invalid code. Excluded + // from the score (never really tested), but reported so the + // count is honest. + ref_set(nocompile, ref_get(nocompile) + 1) + println(" no-compile ${loc} ${label}") + } + } +} + +// Run one CODE operator (find->repl) across the whole SUT: for each of +// its N code occurrences (string literals skipped), write that +// single-site mutant and classify it, anchored to its source line. +_run_operator(sut_path: string, original: string, find: string, repl: string, + label: string, test_path: string, incl: string, bin: string, + killed: ptr, survived: ptr, nocompile: ptr, survivors: ptr) { + base = _basename(sut_path) + n = _count(original, find) + i = 0 + while i < n { + mutant = _replace_nth(original, find, repl, i) + if string.length(mutant) > 0 { + line = _line_at(original, _offset_of_nth(original, find, i)) + fs.write(sut_path, mutant) + _classify("${base}:${line}", label, test_path, incl, bin, killed, survived, nocompile, survivors) + } + i = i + 1 + } +} + +// Run a string-literal operator across the SUT: replace the content of +// each literal matching want_empty (1 = only ""; 0 = only non-empty) +// with `content`, one at a time, and classify. +_run_string_operator(sut_path: string, original: string, want_empty: int, + content: string, label: string, test_path: string, + incl: string, bin: string, killed: ptr, survived: ptr, + nocompile: ptr, survivors: ptr) { + base = _basename(sut_path) + n = _count_strings(original, want_empty) + i = 0 + while i < n { + mutant = _mutate_nth_string(original, i, want_empty, content) + if string.length(mutant) > 0 { + line = _line_at(original, _offset_of_nth_string(original, i, want_empty)) + fs.write(sut_path, mutant) + _classify("${base}:${line}", label, test_path, incl, bin, killed, survived, nocompile, survivors) + } + i = i + 1 + } +} + +// run(sut_path, test_path, lib_dir) — the whole mutation run: baseline +// sanity, every operator, restore, score report. `lib_dir` is the +// module search dir handed to the per-mutant sub-builds via +// AETHER_LIB_DIR (the var `ae` actually honours); pass "" to default +// to the SUT's directory. Returns the SURVIVOR count (0 = every +// compiling mutant was killed), or -1 on abort (unreadable/empty SUT, +// or the unmutated suite failing/not compiling) — a front-end can gate +// its exit code on either. +run(sut_path: string, test_path: string, lib_dir: string) -> int { + incl = lib_dir + if string.equals(incl, "") == 1 { + // default lib_dir = directory of the SUT + last = -1 + p = string.index_of_from(sut_path, "/", 0) + while p >= 0 { + last = p + p = string.index_of_from(sut_path, "/", p + 1) + } + if last >= 0 { + incl = string.substring(sut_path, 0, last) + } else { + incl = "." + } + } + + original, rerr = fs.read(sut_path) + if string.equals(rerr, "") == 0 { + println("error: could not read SUT '${sut_path}': ${rerr}") + return -1 + } + if string.length(original) == 0 { + println("error: SUT '${sut_path}' is empty") + return -1 + } + + // A temp binary path for the per-mutant build+drain oracle, + // pid-suffixed so concurrent runs can't clobber each other. + bin = "/tmp/ae_mutate_probe_${os.getpid()}" + + println("Aether mutation testing (std.mutation)") + println(" SUT: ${sut_path}") + println(" test: ${test_path}") + println("") + + // Sanity: the unmutated suite must PASS (KILLED would mean it's + // already failing; NOCOMPILE means the test doesn't build). + baseline = _oracle(test_path, incl, bin) + if baseline != SURVIVED { + if baseline == NOCOMPILE { + println("ABORT: the unmutated test does not compile (check lib_dir / SUT)") + } else { + println("ABORT: the test suite does not pass on the unmutated SUT") + println(" (fix the suite first; mutation score is meaningless otherwise)") + } + fs.write(sut_path, original) + return -1 + } + println(" baseline: suite passes on unmutated SUT ✓") + println("") + + killed = ref(0) + survived = ref(0) + nocompile = ref(0) + survivors = list.new() + + // Core ~6 operators. Multi-char comparisons first so single-char + // rules can't partially match them. + _run_operator(sut_path, original, " >= ", " < ", "GTE->LT", test_path, incl, bin, killed, survived, nocompile, survivors) + _run_operator(sut_path, original, " <= ", " > ", "LTE->GT", test_path, incl, bin, killed, survived, nocompile, survivors) + _run_operator(sut_path, original, " == ", " != ", "EQ->NE", test_path, incl, bin, killed, survived, nocompile, survivors) + _run_operator(sut_path, original, " != ", " == ", "NE->EQ", test_path, incl, bin, killed, survived, nocompile, survivors) + _run_operator(sut_path, original, " + ", " - ", "ADD->SUB", test_path, incl, bin, killed, survived, nocompile, survivors) + _run_operator(sut_path, original, " - ", " + ", "SUB->ADD", test_path, incl, bin, killed, survived, nocompile, survivors) + _run_operator(sut_path, original, " * ", " / ", "MUL->DIV", test_path, incl, bin, killed, survived, nocompile, survivors) + _run_operator(sut_path, original, " && ", " || ", "AND->OR", test_path, incl, bin, killed, survived, nocompile, survivors) + _run_operator(sut_path, original, " || ", " && ", "OR->AND", test_path, incl, bin, killed, survived, nocompile, survivors) + // Plain-`>` / plain-`<` last (after >= / <= consumed). These still + // can't see a bare `>` that abuts other chars, by design. + _run_operator(sut_path, original, " > ", " < ", "GT->LT", test_path, incl, bin, killed, survived, nocompile, survivors) + _run_operator(sut_path, original, " < ", " > ", "LT->GT", test_path, incl, bin, killed, survived, nocompile, survivors) + + // String-literal mutators. STR->EMPTY blanks a non-empty literal + // (catches tests that don't pin the returned string); EMPTY->NONEMPTY + // fills an empty one with a sentinel (catches the unchecked-empty + // case). Literals inside comments get mutated too — harmless (a + // comment change is a no-op the suite ignores → survives as a known + // false mutant; docs/mutation-testing.md notes this). + _run_string_operator(sut_path, original, 0, "", "STR->EMPTY", test_path, incl, bin, killed, survived, nocompile, survivors) + _run_string_operator(sut_path, original, 1, "AE_MUTANT", "EMPTY->NONEMPTY", test_path, incl, bin, killed, survived, nocompile, survivors) + + // Always restore the original — even though we wrote `original` + // after the last mutant via the loop, make it explicit. + fs.write(sut_path, original) + os.system("rm -f '${bin}'") + + k = ref_get(killed) + s = ref_get(survived) + nc = ref_get(nocompile) + // The score denominator is COMPILING mutants only — a mutant that + // doesn't type-check was never really tested, so counting it as a + // kill would flatter the score (and counting it as a survivor would + // be a false gap). Report it separately. + scored = k + s + println("") + if scored == 0 { + if nc > 0 { + println(" no scorable mutants (${nc} did not compile)") + } else { + println(" no mutation sites found (no padded operators in the SUT)") + } + return 0 + } + score = (k * 100) / scored + println(" ${k}/${scored} mutants killed — mutation score ${score}%") + if nc > 0 { + println(" (${nc} excluded — did not compile)") + } + if s > 0 { + println(" ${s} survived (test gaps):") + m = list.size(survivors) + j = 0 + while j < m { + println(" - ${list.get_raw(survivors, j)}") + j = j + 1 + } + } + return s +} diff --git a/tests/ae_sweep_prune.txt b/tests/ae_sweep_prune.txt index 45c5ece20..8f90a8002 100644 --- a/tests/ae_sweep_prune.txt +++ b/tests/ae_sweep_prune.txt @@ -135,6 +135,7 @@ tests/integration/module_reexport/ tests/integration/module_token_cap/ tests/integration/module_var_cross_import/ tests/integration/multi_tu_import_link/ +tests/integration/mutation_testing/ tests/integration/namespace_ tests/integration/optional_reject/ tests/integration/or_block_reject/ diff --git a/tests/integration/mutation_testing/fixture/sut.ae b/tests/integration/mutation_testing/fixture/sut.ae new file mode 100644 index 000000000..32889d6ad --- /dev/null +++ b/tests/integration/mutation_testing/fixture/sut.ae @@ -0,0 +1,11 @@ +// Regression fixture for std.mutation. Two operator sites, chosen so +// the mutation outcome is fully deterministic. The function `add` is +// tested (its arithmetic mutant gets killed); the function `mul` is not +// tested (its mutant survives). Comments here deliberately avoid any +// whitespace-padded operator token so the text-based mutator sees only +// the two real code sites below. +exports(add, mul) + +add(a: int, b: int) -> int { return a + b } + +mul(a: int, b: int) -> int { return a * b } diff --git a/tests/integration/mutation_testing/fixture/sut_test.ae b/tests/integration/mutation_testing/fixture/sut_test.ae new file mode 100644 index 000000000..5aa4740f2 --- /dev/null +++ b/tests/integration/mutation_testing/fixture/sut_test.ae @@ -0,0 +1,16 @@ +// std.spec test for the regression fixture. Deliberately tests `add` but +// NOT `mul`, so the mutation run yields exactly one killed (ADD->SUB) +// and one survivor (MUL->DIV). The harness pins those numbers. +import std.spec +import sut + +main() { + fw = spec.init() + spec.describe(fw, "sut") { + spec.it("add is tested") callback { + spec.assert_eq(sut.add(2, 3), 5, "2 plus 3") + } + // mul is intentionally NOT tested -> MUL->DIV survives. + } + spec.run_summary(fw) +} diff --git a/tests/integration/mutation_testing/fixture_str/sut.ae b/tests/integration/mutation_testing/fixture_str/sut.ae new file mode 100644 index 000000000..f64acd4de --- /dev/null +++ b/tests/integration/mutation_testing/fixture_str/sut.ae @@ -0,0 +1,14 @@ +// String-mutation regression fixture for std.mutation. Three string +// literals in the code below: name (tested, killed), motto (untested, +// survives), and help, whose body holds an arithmetic operator inside a +// string that the operator mutators MUST skip. No code-level arithmetic +// operators exist, so every mutant is a string mutant: one killed, two +// survived. (This comment avoids quote characters on purpose, so the +// mutator sees only the three real literals below.) +exports(name, motto, help) + +name() -> string { return "alice" } + +motto() -> string { return "untested" } + +help() -> string { return "a + b" } diff --git a/tests/integration/mutation_testing/fixture_str/sut_test.ae b/tests/integration/mutation_testing/fixture_str/sut_test.ae new file mode 100644 index 000000000..a3a3fba7f --- /dev/null +++ b/tests/integration/mutation_testing/fixture_str/sut_test.ae @@ -0,0 +1,14 @@ +import std.spec +import sut + +main() { + fw = spec.init() + spec.describe(fw, "sut") { + // Only name() is pinned. motto() and help() are unchecked, so + // blanking their strings survives. + spec.it("name") callback { + spec.assert_str_eq(sut.name(), "alice", "name") + } + } + spec.run_summary(fw) +} diff --git a/tests/integration/mutation_testing/test_mutation.sh b/tests/integration/mutation_testing/test_mutation.sh new file mode 100755 index 000000000..815182c8a --- /dev/null +++ b/tests/integration/mutation_testing/test_mutation.sh @@ -0,0 +1,92 @@ +#!/bin/sh +# Regression test for the std.mutation mutation-testing driver +# (adopted from aeocha contrib/mutate; run via the runnable front-end +# at examples/mutation-testing/mutate.ae). +# +# Runs the driver against a deterministic fixture (fixture/sut.ae + +# sut_test.ae) where exactly two operator sites exist: +# - `add` is tested -> ADD->SUB mutant KILLED +# - `mul` is NOT tested -> MUL->DIV mutant SURVIVES +# so the run must report exactly "1/2 ... 50%" with a MUL->DIV survivor. +# +# Asserts four things: +# 1. baseline (unmutated suite) passes, +# 2. the exact mutation score line, +# 3. the expected survivor is listed, +# 4. the SUT is restored byte-identical (md5 before == after) — the +# safety-critical property; a driver that corrupts source is worse +# than no driver. +# +# POSIX-only (the driver shells out via rm/ae and the oracle assumes +# /bin/sh); skipped on Windows and where `ae` is not built. + +set -e + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)" +AE="$ROOT/build/ae" +SUT="$SCRIPT_DIR/fixture/sut.ae" +TEST="$SCRIPT_DIR/fixture/sut_test.ae" + +if [ "$OS" = "Windows_NT" ]; then + echo " [SKIP] mutation_testing: driver shells out via POSIX rm/ae" + exit 0 +fi + +if [ ! -x "$AE" ]; then + echo " [SKIP] mutation_testing: ae not built" + exit 0 +fi + +if ! command -v md5sum >/dev/null; then + MD5="cksum" # fallback; any stable hash works for before==after +else + MD5="md5sum" +fi + +cd "$ROOT" || exit 1 + +TMPDIR="$(mktemp -d)" +trap 'rm -rf "$TMPDIR"' EXIT + +fail() { + echo " [FAIL] mutation_testing — $1" + [ -f "$TMPDIR/out.log" ] && tail -40 "$TMPDIR/out.log" | sed 's/^/ /' + exit 1 +} + +# Hash the SUT before the run (property 4). +SUT_BEFORE="$($MD5 "$SUT" | awk '{print $1}')" + +rm -rf "$HOME/.aether/cache" +# AE_BIN points the driver's per-mutant sub-builds at the in-tree ae; +# AETHER_HOME resolves std.* (std.mutation, std.spec) from this tree. +# The lib_dir arg (the fixture dir) is where `import sut` resolves. +if ! AETHER_HOME="$ROOT" AE_BIN="$AE" \ + "$AE" run "$ROOT/examples/mutation-testing/mutate.ae" -- \ + "$SUT" "$TEST" "$SCRIPT_DIR/fixture" >"$TMPDIR/out.log" 2>&1; then + fail "driver exited non-zero" +fi + +# Property 4 first: SUT must be unchanged regardless of outcome. +SUT_AFTER="$($MD5 "$SUT" | awk '{print $1}')" +if [ "$SUT_BEFORE" != "$SUT_AFTER" ]; then + fail "SUT not restored (md5 $SUT_BEFORE -> $SUT_AFTER)" +fi + +# Property 1: baseline passed. +grep -q "baseline: suite passes" "$TMPDIR/out.log" || fail "no baseline-pass line" + +# Property 2: exact score. +grep -q "1/2 mutants killed — mutation score 50%" "$TMPDIR/out.log" \ + || fail "unexpected mutation score (wanted 1/2, 50%)" + +# Property 3: the known survivor and kill are reported, each anchored to a +# source location (file:line). `mul` is on line 11, `add` on line 9. +grep -Eq "SURVIVED +sut\.ae:11 +MUL->DIV" "$TMPDIR/out.log" \ + || fail "MUL->DIV survivor not reported at sut.ae:11" +grep -Eq "killed +sut\.ae:9 +ADD->SUB" "$TMPDIR/out.log" \ + || fail "ADD->SUB not killed at sut.ae:9" + +echo " [PASS] mutation_testing" +exit 0 diff --git a/tests/integration/mutation_testing/test_mutation_strings.sh b/tests/integration/mutation_testing/test_mutation_strings.sh new file mode 100755 index 000000000..c676e7857 --- /dev/null +++ b/tests/integration/mutation_testing/test_mutation_strings.sh @@ -0,0 +1,79 @@ +#!/bin/sh +# Regression for std.mutation's STRING-literal mutation + the +# operator-in-string skip (string-boundary awareness). +# +# fixture_str/sut.ae has three string literals and one of them (help) +# contains an arithmetic operator INSIDE the string ("a + b"): +# - name() is tested -> STR->EMPTY killed +# - motto() is NOT tested -> STR->EMPTY survives +# - help() is NOT tested -> STR->EMPTY survives +# and there are NO code-level arithmetic operators, so the run must be +# exactly "1/3 ... 33%" with two STR->EMPTY survivors. +# +# Asserts: +# 1. baseline passes, +# 2. exact score "1/3 ... 33%", +# 3. a STR->EMPTY survivor is reported (string mutation works), +# 4. NO ADD->SUB mutant appears — the ` + ` inside help()'s string was +# skipped (the boundary-awareness / false-mutant fix), +# 5. SUT restored byte-identical. + +set -e + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)" +AE="$ROOT/build/ae" +SUT="$SCRIPT_DIR/fixture_str/sut.ae" +TEST="$SCRIPT_DIR/fixture_str/sut_test.ae" + +if [ "$OS" = "Windows_NT" ]; then + echo " [SKIP] mutation_testing_strings: driver shells out via POSIX rm/ae" + exit 0 +fi +if [ ! -x "$AE" ]; then + echo " [SKIP] mutation_testing_strings: ae not built" + exit 0 +fi +if command -v md5sum >/dev/null; then MD5="md5sum"; else MD5="cksum"; fi + +cd "$ROOT" || exit 1 +TMPDIR="$(mktemp -d)" +trap 'rm -rf "$TMPDIR"' EXIT + +fail() { + echo " [FAIL] mutation_testing_strings — $1" + [ -f "$TMPDIR/out.log" ] && tail -40 "$TMPDIR/out.log" | sed 's/^/ /' + exit 1 +} + +SUT_BEFORE="$($MD5 "$SUT" | awk '{print $1}')" + +rm -rf "$HOME/.aether/cache" +# AE_BIN points the driver's per-mutant sub-builds at the in-tree ae; +# AETHER_HOME resolves std.*; lib_dir arg is where `import sut` resolves. +if ! AETHER_HOME="$ROOT" AE_BIN="$AE" \ + "$AE" run "$ROOT/examples/mutation-testing/mutate.ae" -- \ + "$SUT" "$TEST" "$SCRIPT_DIR/fixture_str" >"$TMPDIR/out.log" 2>&1; then + fail "driver exited non-zero" +fi + +SUT_AFTER="$($MD5 "$SUT" | awk '{print $1}')" +[ "$SUT_BEFORE" = "$SUT_AFTER" ] || fail "SUT not restored (md5 $SUT_BEFORE -> $SUT_AFTER)" + +grep -q "baseline: suite passes" "$TMPDIR/out.log" || fail "no baseline-pass line" +grep -q "1/3 mutants killed — mutation score 33%" "$TMPDIR/out.log" \ + || fail "unexpected score (wanted 1/3, 33%)" +# Survivors/kill are anchored to source lines: name() killed on line 10, +# motto() survives on line 12. +grep -Eq "SURVIVED +sut\.ae:12 +STR->EMPTY" "$TMPDIR/out.log" \ + || fail "no STR->EMPTY survivor at sut.ae:12 (string mutation broken?)" +grep -Eq "killed +sut\.ae:10 +STR->EMPTY" "$TMPDIR/out.log" \ + || fail "STR->EMPTY not killed at sut.ae:10" +# The crucial boundary-awareness check: the ` + ` inside help()'s string +# must NOT have produced an operator mutant. +if grep -q "ADD->SUB" "$TMPDIR/out.log"; then + fail "ADD->SUB appeared — operator inside a string literal was NOT skipped" +fi + +echo " [PASS] mutation_testing_strings" +exit 0