diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 100644 index 0000000..41f7433 --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1,32 @@ +# Copilot Instructions for multiplex + +*multiplex* is a modular framework for prototyping LLM-based mutation testing. One run mutates a single Java method (located via tree-sitter), generates mutants with an LLM, splices each mutant back into the source file, and runs the target project's tests to see which mutants are killed. + +## Read the docs before reading source + +Detailed, task-scoped documentation lives in `docs/` — load only the file relevant to the task: + +- `docs/ARCHITECTURE.md` — end-to-end pipeline, approaches, execution backends, output artifacts. +- `docs/MODULE_REFERENCE.md` — every function's signature, behavior, and side effects. +- `docs/CONFIG.md` — full `config.yml` schema. +- `docs/EXTENDING.md` — checklists for adding approaches, execution backends, or LLM providers. +- `docs/DEVELOPMENT.md` — setup, conventions, CI, and the current list of known bugs/gotchas. + +## Commands + +```bash +uv sync -p 3.13 # install deps (pin 3.13: tiktoken fails to build on 3.14) +uv run pytest tests # all tests — must run from repo root (relative resource paths) +uv run pytest tests/checks/test_compiles.py::test_name # one test +uv run ruff check . # lint (matches CI) +uv run ./multiplex ./path/to/config.yml # run the tool +``` + +## Rules that prevent broken changes + +- **Import convention**: the tool runs as a directory (`uv run ./multiplex`), so `multiplex/` itself is on `sys.path`. Modules inside `multiplex/` import each other **without** the package prefix (`from model import Model`, `from util.io import ...`); tests import **with** it (`from multiplex.checks... import ...`). Match the style of the file being edited. +- Adding a mutant-generation approach touches three places: a new `multiplex/approach//` package with `controller.py`, an `APPROACH_PROMPT_KEYS` entry in `multiplex/prompts.py`, and a dispatch branch in `multiplex/__main__.py`. Only the selected approach's `system_prompts` keys are required; `resolve_prompts` validates the approach and its keys up front, raising `SystemExit` before any destructive step. +- Mutant files must be complete replacement methods written to `output/-mutants/mutant_N.java`; they are spliced verbatim over the original method's byte range. +- tree-sitter versions are pinned (`tree-sitter==0.23.2`, `tree-sitter-java==0.23.5`); do not bump them casually — the parsing code depends on that API. +- A runnable end-to-end example lives in `examples/` (self-contained Maven project, `basic` approach, `mvn` backend): `uv run multiplex ./examples/config.yml`. The STPA mutant loop still drops its last item — check `docs/DEVELOPMENT.md` § Known issues. +- Keep pure logic (parsing, splicing, checks) separate from `model.make_request` call sites — LLM calls are not mocked in tests. diff --git a/.gitignore b/.gitignore index 70c19f7..393bc5c 100644 --- a/.gitignore +++ b/.gitignore @@ -175,5 +175,9 @@ cython_debug/ .idea/* .vscode/* -# OS -.DS_Store \ No newline at end of file +# OS +.DS_Store + +# multiplex run artifacts (created under at runtime) +output/ +*.orig \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md index 5e264f0..3376c48 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,6 +27,7 @@ uv run ./multiplex ./path/to/config.yml # run the tool ## Critical rules - **Import convention**: modules inside `multiplex/` import each other **without** the package prefix (`from model import Model`); tests import **with** it (`from multiplex.checks... import ...`). Match the file you are editing. Rationale in `docs/DEVELOPMENT.md`. -- Adding an approach touches three places (package + prompt keys + dispatch); all `system_prompts` config keys are read unconditionally at startup. See `docs/EXTENDING.md`. +- Adding an approach touches three places (package + `APPROACH_PROMPT_KEYS` entry in `multiplex/prompts.py` + dispatch branch). Only the selected approach's `system_prompts` keys are required, validated up front by `resolve_prompts`. See `docs/EXTENDING.md`. - tree-sitter versions are pinned; do not bump them casually. -- Check `docs/DEVELOPMENT.md` § Known issues before building on `execute/maven.py` (broken) or the STPA mutant loop (off-by-one); `execute/defects4j.py` is the reference execution backend. +- A runnable example lives in `examples/`: `uv run multiplex ./examples/config.yml` (self-contained Maven project, needs `mvn` + a JDK + a running Ollama). See `docs/DEVELOPMENT.md` § Example. +- The STPA mutant loop has a verified off-by-one — see `docs/DEVELOPMENT.md` § Known issues. diff --git a/CLAUDE.md b/CLAUDE.md index bf61101..0a58995 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -29,7 +29,8 @@ uv run ./multiplex ./path/to/config.yml # run the tool ## Critical conventions - **Import convention**: the tool runs as a directory (`uv run ./multiplex`), so `multiplex/` itself is on `sys.path`. Modules inside `multiplex/` import each other **without** the package prefix (`from model import Model`, `from util.io import ...`); tests import **with** it (`from multiplex.checks... import ...`). Match the style of the file you are editing. -- Pipeline entry point and dispatch: `multiplex/__main__.py`. Adding an approach touches three places (new `approach//controller.py` package, prompt keys in config **and** the `prompts` dict in `__main__.py`, dispatch branch) — see docs/EXTENDING.md. -- All `system_prompts` config keys are read unconditionally at startup; a missing key is an immediate KeyError. +- Pipeline entry point and dispatch: `multiplex/__main__.py`. Adding an approach touches three places (new `approach//controller.py` package, a `APPROACH_PROMPT_KEYS` entry in `multiplex/prompts.py`, dispatch branch in `__main__.py`) — see docs/EXTENDING.md. +- Only the selected approach's `system_prompts` keys are required; `resolve_prompts` in `multiplex/prompts.py` validates them (and the approach name) up front, raising `SystemExit` before any destructive step. - `/output/` is wiped at the start of every run; the target source file is edited in place and restored from a `.orig` backup. -- Before relying on `execute/maven.py` or the STPA mutant loop, check docs/DEVELOPMENT.md § Known issues — both have verified bugs (Maven backend is broken; use `execute/defects4j.py` as reference). +- A runnable end-to-end example lives in `examples/` (self-contained Maven project, `basic` approach, `mvn` backend): `uv run multiplex ./examples/config.yml` (needs `mvn`, a JDK, and a running Ollama). See docs/DEVELOPMENT.md § Example. +- The STPA mutant loop has a verified off-by-one (drops its last UCA) — see docs/DEVELOPMENT.md § Known issues. diff --git a/README.md b/README.md index 7865e35..d4c66ab 100644 --- a/README.md +++ b/README.md @@ -51,7 +51,7 @@ Ensure Ollama is running when using *multiplex* for this example. llm: model: ollama/gpt-oss:20b endpoint: http://127.0.0.1:11434 - tokenenvvar: # env var name storing token (NOT TOKEN) + token_env_var: # env var name storing token (NOT TOKEN) ``` [IMPORTANT] Security Note: Never hardcode API keys in your config file. Set an environment variable and reference its name in the tokenenvvar field. @@ -68,6 +68,18 @@ Once configured and modules are set up, users can run *multiplex* using the foll uv run /path/to/multiplex ./path/to/config.yml ``` +#### Try the bundled example +A self-contained example (a small Maven project mutated with the `basic` +approach) is included. With `mvn` + a JDK on your `PATH` and a running Ollama +(`ollama pull gpt-oss:20b`, or edit `llm.model` in the config), run from the +repo root: +```bash +uv run multiplex ./examples/config.yml +``` +Results are written to `examples/project/example/output/basic-mutants/` +(`mutant_summary.csv`). See [`examples/README.md`](examples/README.md) for +details. + ## 🧩 Existing Modules *multiplex* currently includes five mutant generation modules and two execution and evaluation modules: diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 04573d6..d17a447 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -74,8 +74,12 @@ Selected by `project.runtool`: each compilable mutant's tests with a timeout of 5× baseline. A surviving mutant is one whose test run reports `Failing tests: 0`. Requires the `defects4j` CLI and a `JDK_11` env var (see DEVELOPMENT.md). -- `mvn` → `maven.py` — **currently broken/incomplete**; do not rely on it - (details in DEVELOPMENT.md § Known issues). +- `mvn` → `maven.py` — self-contained backend for plain Maven projects. Runs + `mvn -f clean test`; a mutant survives if the build passes + (`BUILD SUCCESS`). Baselines the original first (aborts if its tests fail), + then per mutant does the same equivalence → rewrite → compilable → test → + summary flow as the Defects4J backend. Used by the runnable example under + `examples/` (see DEVELOPMENT.md § Example). ## Output artifacts diff --git a/docs/CONFIG.md b/docs/CONFIG.md index 30a35da..da13225 100644 --- a/docs/CONFIG.md +++ b/docs/CONFIG.md @@ -4,8 +4,10 @@ `uv run ./multiplex ./path/to/config.yml`. Template: [`examples/config.yml`](../examples/config.yml). -All keys below are read unconditionally by `multiplex/__main__.py` — a missing -key raises `KeyError` at startup, even for approaches you are not running. +Only the keys the selected `mutation.approach` needs are required. `project`, +`mutation`, and `llm` keys are always required; `system_prompts` keys are +required per-approach (see that section). Missing required keys fail fast at +startup with an actionable `SystemExit`, before any output is wiped. ## `project` @@ -15,13 +17,13 @@ key raises `KeyError` at startup, even for approaches you are not running. | `filename` | path | The `.java` source file containing the method under test. Backed up to `.orig` during the run. | | `method` | string | Name of the method (or constructor) to mutate. | | `line` | int | 1-based line number of the method **name identifier** in `filename` (not annotations above it). Both `method` and `line` must match for extraction to succeed; disambiguates overloads. | -| `runtool` | `mvn` \| `d4j` | Execution backend. `d4j` (Defects4J) is the complete one; `mvn` is currently broken (DEVELOPMENT.md § Known issues). | +| `runtool` | `mvn` \| `d4j` | Execution backend. `mvn` runs a plain Maven project (`mvn clean test`) and is used by the runnable `examples/` setup; `d4j` targets a Defects4J checkout (needs the `defects4j` CLI + `JDK_11`). | ## `mutation` | Key | Type | Meaning | |-----|------|---------| -| `approach` | `basic` \| `hazop` \| `stpa` \| `mutahunter` \| `llmorpheus` | Mutant-generation approach. Any other value prints "Invalid approach" and continues to the execution phase (which then fails on the missing mutants dir). | +| `approach` | `basic` \| `hazop` \| `stpa` \| `mutahunter` \| `llmorpheus` | Mutant-generation approach. An unknown value fails fast at startup with a `SystemExit` listing the valid approaches. | ## `llm` @@ -36,8 +38,12 @@ example config use `token_env_var`. `token_env_var` is correct. ## `system_prompts` -All nine keys are required (multi-line YAML strings, typically `|` blocks). -User prompts are constructed in code; only system prompts live in config. +Multi-line YAML strings (typically `|` blocks). Only the keys used by the +selected `mutation.approach` are required — prompts for other approaches may be +omitted. The approach→required-keys mapping is `APPROACH_PROMPT_KEYS` in +`multiplex/prompts.py`; a missing or empty required key raises a `SystemExit` +naming it. User prompts are constructed in code; only system prompts live in +config. | Key | Used by | |-----|---------| diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md index 7b1ac03..3cc9caa 100644 --- a/docs/DEVELOPMENT.md +++ b/docs/DEVELOPMENT.md @@ -24,6 +24,30 @@ uv run ./multiplex ./path/to/config.yml # run the tool CI (GitHub Actions, on push/PR to `main`): `test.yml` runs `uv run pytest tests` on Python 3.13; `ruff.yml` runs `uv run ruff check`. +## Example + +A self-contained end-to-end example lives under `examples/`: + +```bash +uv run multiplex ./examples/config.yml # from the repo root +``` + +- Target: `examples/project/example/` — a tiny Maven project with one method + (`com.example.Classifier.classify`) and JUnit tests pinning its behavior. +- `examples/config.yml` uses the `basic` approach and the `mvn` runtool. +- Prerequisites: `mvn` + a JDK 11+ on PATH (the project targets Java 11; if + `mvn` picks up an older JDK via `JAVA_HOME` the compile fails with "release + version 11 not supported"), and a running Ollama serving the model + named in `llm.model` (default `gpt-oss:20b`; point it at any model you have, + e.g. `ollama pull gpt-oss:20b`). Any LiteLLM-supported endpoint works if you + edit `llm`. +- Output lands in `examples/project/example/output/basic-mutants/` + (`mutant_N.java` plus `mutant_summary.csv`). Run artifacts (`output/`, + `*.orig`, Maven `target/`) are git-ignored. +- The `basic` approach makes 10 LLM calls and the `mvn` backend runs the test + suite once per compilable mutant, so a full run takes a few minutes (longer + on a slow/local model). + ## Import convention (critical) The tool is executed as a directory (`uv run ./multiplex`), which puts @@ -58,21 +82,28 @@ same process; keep the two worlds separate. Bugs — fix or work around, don't replicate: -1. **`execute/maven.py` is broken**: `run_mutants` calls `rewrite_method` with - 5 args (signature takes 4 → TypeError), never calls its own - `_execute_mutant` (so `mvn test` never runs), and passes a directory to - `check_mutant_compilable` (which parses a single file). Use - `execute/defects4j.py` as the reference implementation. -2. **STPA off-by-one**: `approach/stpa/code_generator.py` loops +1. **STPA off-by-one**: `approach/stpa/code_generator.py` loops `range(0, ucas_count - 1)`, silently skipping the last UCA. -3. **Method-not-found crash**: `extract_method_from_file` returns `None` on a +2. **Method-not-found crash**: `extract_method_from_file` returns `None` on a miss, but `__main__.py` unpacks it as a tuple → opaque TypeError. Symptom: check `project.method`/`project.line` in the config (`line` must be the line of the method-name identifier). -4. **README key mismatch**: README shows `llm.tokenenvvar`; the code reads +3. **README key mismatch**: README shows `llm.tokenenvvar`; the code reads `llm.token_env_var` (the example config is correct). -5. **`tests/execute/test_defects4j.py` is an empty stub** — the execute layer - has zero test coverage. +4. **`tests/execute/test_defects4j.py` is an empty stub** — the Defects4J + backend has no test coverage. (The Maven backend is covered by + `tests/execute/test_maven.py`, which mocks `mvn` via `_execute`; that file's + `conftest.py` adds `multiplex/` to `sys.path` so the flat-import backend + module is importable — the pattern to copy for a Defects4J test.) + +Recently fixed (kept here so stale references elsewhere are recognizable): +- The `prompts` dict in `__main__.py` used to read **all** `system_prompts` + keys unconditionally, so any missing key crashed every run with `KeyError` — + the shipped example config itself crashed this way. Now only the selected + approach's keys are required, validated up front via `multiplex/prompts.py`. +- `execute/maven.py` used to be broken (wrong-arity `rewrite_method` call, never + ran `mvn test`, passed a directory to `check_mutant_compilable`). It now + mirrors `execute/defects4j.py` and drives the runnable example (§ Example). Behavioral gotchas — by design (or at least current design), be aware: @@ -85,9 +116,10 @@ Behavioral gotchas — by design (or at least current design), be aware: restore). - "Compilable" means *parses without tree-sitter ERROR nodes* — no compiler runs; type errors and unresolved symbols count as compilable. -- An invalid `mutation.approach` value only prints "Invalid approach" and - falls through to the execution phase. -- All nine `system_prompts` keys are read at startup regardless of the chosen - approach — a missing key is a KeyError before anything runs. +- An invalid `mutation.approach` value, or a missing/empty required + `system_prompts` key for the chosen approach, raises `SystemExit` at startup + (before the output wipe). Only the selected approach's prompt keys are + required — see `multiplex/prompts.py`. - Mutahunter prompts are not bundled (licensing); the user pastes them into - the config. + the config. Running the `mutahunter` approach without them fails fast with a + clear message. diff --git a/docs/EXTENDING.md b/docs/EXTENDING.md index dbe1494..5631836 100644 --- a/docs/EXTENDING.md +++ b/docs/EXTENDING.md @@ -29,11 +29,12 @@ the multi-prompt-chain template. mutant = mutant.removeprefix("```java") mutant = mutant.split("```", 1)[0] ``` -2. **Register the prompt key(s)**: add `_...` entries to the `prompts` - dict in `multiplex/__main__.py` *and* to the `system_prompts` section of - every config file (including `examples/config.yml`). All keys in that dict - are read unconditionally — omitting one breaks **every** run with KeyError, - not just runs of your approach. +2. **Register the prompt key(s)**: add a `"": [...]` entry to + `APPROACH_PROMPT_KEYS` in `multiplex/prompts.py` listing the + `system_prompts` keys your approach reads. Only the selected approach's keys + are required at runtime, so users running other approaches need not define + yours (and vice versa). Document the keys in `docs/CONFIG.md`; adding them to + `examples/config.yml` (a commented stub is fine) is optional but helpful. 3. **Register the dispatch branch**: add an `elif config['mutation']['approach'] == "":` branch in `multiplex/__main__.py` calling `.main(model, output_path, prompts)`, plus the corresponding @@ -44,7 +45,8 @@ the multi-prompt-chain template. ## Add an execution/evaluation backend -Model on `multiplex/execute/defects4j.py` (not `maven.py`, which is broken). +Model on `multiplex/execute/defects4j.py` or `multiplex/execute/maven.py` +(both follow the same flow; `maven.py` is the simpler, self-contained one). 1. Create `multiplex/execute/.py` exposing: ```python diff --git a/docs/MODULE_REFERENCE.md b/docs/MODULE_REFERENCE.md index 10370f7..b66a66d 100644 --- a/docs/MODULE_REFERENCE.md +++ b/docs/MODULE_REFERENCE.md @@ -20,6 +20,20 @@ class Model: - If `"azure" in endpoint`, sets `AZURE_AI_API_BASE` and `AZURE_AI_API_KEY`. - `messages` is a LiteLLM/OpenAI-style list of `{"role": ..., "content": ...}`. +## prompts.py + +```python +APPROACH_PROMPT_KEYS: dict[str, list[str]] # approach name -> required system_prompts keys +resolve_prompts(config, approach) -> dict # {key: prompt} for the approach's keys +``` + +- `resolve_prompts` reads only the selected approach's keys from + `config["system_prompts"]`. Raises `SystemExit` (actionable message) if + `approach` is unknown or any required key is missing/empty. Pure function, no + I/O — called by `__main__.py` before any destructive step. Importable in tests + as `from multiplex.prompts import ...` (no sibling imports, so it works under + both the flat runtime import and the package-prefixed test import). + ## util/io.py ```python @@ -162,7 +176,12 @@ run_mutants(project_root, filename, output_path, method_start_byte, method_end_byte, duplicate, approach) ``` -- **Broken — do not use as reference.** Calls `rewrite_method` with a stale - 5-argument signature (TypeError), never invokes its own `_execute_mutant` - (`mvn clean test`), and calls `check_mutant_compilable` on a directory. - Model a fix on `execute/defects4j.py`. See DEVELOPMENT.md § Known issues. +- `_execute(project_root) -> bool`: runs `mvn -f clean test`; + True if the output contains `BUILD SUCCESS` (all tests green → mutant + survived), False otherwise (mutant killed, incl. compile failures Maven + catches that tree-sitter does not). +- `run_mutants`: same flow as the Defects4J backend — baseline the original + project (raises `IOError` if its tests fail), then per mutant: restore from + backup → equivalence check → rewrite → compilable check → (if compilable) run + tests → append `[name, equivalent, compilable, survives]` → write + `mutant_summary.csv`. Drives the runnable example under `examples/`. diff --git a/examples/README.md b/examples/README.md new file mode 100644 index 0000000..19b6713 --- /dev/null +++ b/examples/README.md @@ -0,0 +1,51 @@ +# Example run + +A self-contained, end-to-end example of *multiplex*. + +```bash +# from the repo root +uv run multiplex ./examples/config.yml +``` + +## What it does + +Mutates one method — `com.example.Classifier.classify` in the small Maven +project under [`project/example/`](project/example) — using the **basic** +approach, then runs that project's JUnit tests against each mutant with the +**mvn** backend to see which mutants survive. + +## Prerequisites + +- `mvn` and a JDK **11 or newer** on your `PATH` (the `mvn` runtool runs + `mvn clean test`; the project targets Java 11). If `mvn` uses an older JDK — + e.g. via `JAVA_HOME` — the build fails with "release version 11 not + supported"; point `JAVA_HOME` at a JDK 11+ or lower `maven.compiler.release` + in `project/example/pom.xml`. +- A running [Ollama](https://ollama.com) serving the model named in + `config.yml` under `llm.model` (default `gpt-oss:20b`): + + ```bash + ollama pull gpt-oss:20b # or edit llm.model to a model you already have + ``` + + Any LiteLLM-supported provider works — edit the `llm` section of + `config.yml` (`model`, `endpoint`, `token_env_var`). + +## Output + +Written under `project/example/output/basic-mutants/`: + +- `mutant_N.java` — each generated mutant (the mutated method body). +- `mutant_summary.csv` — one row per mutant with columns + `MUTANT, EQUIVALENCE, COMPILABLE, SURVIVES`. A surviving mutant is one the + test suite failed to catch. + +These run artifacts (plus the `*.orig` source backup and Maven `target/`) are +git-ignored. + +## Trying another approach + +Prompt sets are optional per approach, so you can switch `mutation.approach` in +`config.yml` to `hazop` or `stpa` without editing anything else — their prompts +are already included. `mutahunter` and `llmorpheus` need their own prompts added +first (see the comments in `config.yml` and the top-level README). diff --git a/examples/config.yml b/examples/config.yml index f2c8256..2e2043d 100644 --- a/examples/config.yml +++ b/examples/config.yml @@ -1,21 +1,39 @@ -# config.yml +# config.yml — self-contained example. +# Run from the repo root: uv run multiplex ./examples/config.yml +# +# Requirements for this example: +# - Maven (`mvn`) and a JDK 11+ on PATH (used by the `mvn` runtool below; +# the example project targets Java 11 — see project/example/pom.xml). +# - A running Ollama serving the model named under `llm.model` +# (`ollama pull gpt-oss:20b`, or point `llm.model` at any model you have). project: - projectroot: ../project/csv_3_fixed - filename: ../project/csv_3_fixed/src/main/java/org/apache/commons/csv/Lexer.java - method: readEscape - line: 87 - runtool: d4j + projectroot: examples/project/example + filename: examples/project/example/src/main/java/com/example/Classifier.java + method: classify + line: 6 + runtool: mvn mutation: - approach: stpa + approach: basic llm: model: ollama/gpt-oss:20b endpoint: http://127.0.0.1:11434 - token_env_var: # env var name storing token (NOT TOKEN) + token_env_var: # env var name storing token (NOT the token itself) system_prompts: + # Only the prompts for the selected `mutation.approach` are required. + # `basic` (the approach used above) needs only `basic_generate_mutants`. + basic_generate_mutants: | + You are a code generator. + When given a Java method, re-implement it with a single small behavioral defect (a mutant). + Change exactly one thing: an operator, a boundary condition, a constant, or a returned value. + Include a concise code comment describing the injected defect. + Output the defective Java code only and nothing else. + + # The prompts below let you switch `mutation.approach` to hazop or stpa + # without editing anything else. hazop_describe_process: | When given a Java method, describe the intent of the code in a line-by-line manner, as concisely as possible. @@ -24,12 +42,12 @@ system_prompts: Break the explanation into line-numbered steps and return the line-numbered steps. hazop_identify_deviations: | For each numbered description, output one version of the string where one of the rules has changed. - The rule must be changed using one of the "guidewords" from HAZOP. + The rule must be changed using one of the "guidewords" from HAZOP. The guidewords and their interpretation is delimited by ###, in the form guideword - interpretation. That is, everything before - is the guideword and everything after is the meaning. - + ### - + No (not, none) - None of the design intent is achieved More (more of, higher) - Quantitative increase in a parameter Less (less of, lower) - Quantitative decrease in a parameter @@ -37,15 +55,15 @@ system_prompts: Part of - Only some of the design intention is achieved Reverse - Logical opposite of the design intent occurs Other than (other) - Complete substitution (another activity takes place or an unusual activity occurs or uncommon condition exists) - + ### - + Please output each new rule, as a csv line, without headings in the format: "number, original_rule, guideword (CAPITALIZED), changed_rule" hazop_implement_deviations: | - You are a code generator. + You are a code generator. When given a description, re-implement the method but with the incorrect behavior/defect as described in the description. - + Include a concise code comment to describe the implemented defect. Output the defective Java code only and nothing else. stpa_describe_control_flow: | @@ -55,7 +73,7 @@ system_prompts: Output the control diagram only, and nothing else. stpa_identify_ucas: | Identify 10 different Unsafe Control Actions (UCAs) within the control diagram described in DOT language, referencing the code, using each of the guidewords delimited by ### below: - + ### * provides * does not provide @@ -64,20 +82,22 @@ system_prompts: * out of order * stopped too soon * applied for too long - + ### - + You must describe each UCA in the format: - + An example of this is: variable x calculated out of order when sorting has not occurred. - + Output the numbered list of UCAs with no empty lines, and nothing else. stpa_implement_ucas: | - You are a code generator. + You are a code generator. When given a description, re-implement the method but with the incorrect behavior/defect as described in the description. - + Include a concise code comment to describe the implemented defect. Output the defective Java code only and nothing else. -NOTE: USER MUST COLLECT MUTAHUNTER PROMPTS THEMSELVES DUE TO LICENSING \ No newline at end of file + # The mutahunter and llmorpheus approaches need their own prompts to run: + # - mutahunter: prompts must be supplied by you (licensing) — see README. + # - llmorpheus: provide `llmorpheus_system` with your mutation instructions. diff --git a/examples/project/example/pom.xml b/examples/project/example/pom.xml new file mode 100644 index 0000000..89dfc39 --- /dev/null +++ b/examples/project/example/pom.xml @@ -0,0 +1,36 @@ + + + 4.0.0 + + com.example + example + 1.0.0 + jar + + + 11 + UTF-8 + 5.10.2 + + + + + org.junit.jupiter + junit-jupiter + ${junit.version} + test + + + + + + + org.apache.maven.plugins + maven-surefire-plugin + 3.2.5 + + + + diff --git a/examples/project/example/src/main/java/com/example/Classifier.java b/examples/project/example/src/main/java/com/example/Classifier.java new file mode 100644 index 0000000..eacfbc5 --- /dev/null +++ b/examples/project/example/src/main/java/com/example/Classifier.java @@ -0,0 +1,14 @@ +package com.example; + +/** Classifies integers by sign. Target of the multiplex example run. */ +public class Classifier { + + public String classify(int n) { + if (n > 0) { + return "positive"; + } else if (n < 0) { + return "negative"; + } + return "zero"; + } +} diff --git a/examples/project/example/src/test/java/com/example/ClassifierTest.java b/examples/project/example/src/test/java/com/example/ClassifierTest.java new file mode 100644 index 0000000..19ece7f --- /dev/null +++ b/examples/project/example/src/test/java/com/example/ClassifierTest.java @@ -0,0 +1,25 @@ +package com.example; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +import org.junit.jupiter.api.Test; + +class ClassifierTest { + + private final Classifier classifier = new Classifier(); + + @Test + void classifiesPositive() { + assertEquals("positive", classifier.classify(5)); + } + + @Test + void classifiesNegative() { + assertEquals("negative", classifier.classify(-5)); + } + + @Test + void classifiesZero() { + assertEquals("zero", classifier.classify(0)); + } +} diff --git a/multiplex/__main__.py b/multiplex/__main__.py index 0845214..5a24a8f 100644 --- a/multiplex/__main__.py +++ b/multiplex/__main__.py @@ -12,6 +12,7 @@ from execute import maven, defects4j from model import Model +from prompts import resolve_prompts from util.io import reset_source_code from util.extract_method import extract_method_from_file @@ -44,17 +45,11 @@ def main(): output_path = Path(config['project']['projectroot'], 'output/') duplicate_file_path = Path(config['project']['filename'] + ".orig") - prompts = { - "hazop_describe_process" : config['system_prompts']['hazop_describe_process'], - "hazop_identify_deviations" : config['system_prompts']['hazop_identify_deviations'], - "hazop_implement_deviations" : config['system_prompts']['hazop_implement_deviations'], - "stpa_describe_control_flow" : config['system_prompts']['stpa_describe_control_flow'], - "stpa_identify_ucas" : config['system_prompts']['stpa_identify_ucas'], - "stpa_implement_ucas" : config['system_prompts']['stpa_implement_ucas'], - "basic_generate_mutants" : config['system_prompts']['basic_generate_mutants'], - "mutahunter_generate_mutants" : config['system_prompts']['mutahunter_generate_mutants'], - "llmorpheus_system" : config['system_prompts']['llmorpheus_system'] - } + + # Validate the approach and its required prompts before any destructive work + # (output wipe, source reset) below. + approach = config['mutation']['approach'] + prompts = resolve_prompts(config, approach) print(f"File: {config['project']['filename']}") print(f"Method: {config['project']['method']}") @@ -81,18 +76,16 @@ def main(): model = Model(model=config['llm']['model'], endpoint=config['llm']['endpoint'], api_key_var=config['llm']['token_env_var']) - if config['mutation']['approach'] == "stpa": + if approach == "stpa": stpa.main(model, output_path, prompts) - elif config['mutation']['approach'] == "hazop": + elif approach == "hazop": hazop.main(model, output_path, prompts) - elif config['mutation']['approach'] == "basic": + elif approach == "basic": basic.main(model, output_path, prompts) - elif config['mutation']['approach'] == "mutahunter": + elif approach == "mutahunter": mutahunter.main(model, output_path, prompts) - elif config['mutation']['approach'] == "llmorpheus": + elif approach == "llmorpheus": llmorpheus.main(model, output_path, prompts) - else: - print("Invalid approach") if config['project']['runtool'] == "mvn": maven.run_mutants( diff --git a/multiplex/execute/maven.py b/multiplex/execute/maven.py index 66caacc..1ed2714 100644 --- a/multiplex/execute/maven.py +++ b/multiplex/execute/maven.py @@ -5,54 +5,76 @@ from pathlib import Path from checks.compilable import check_mutant_compilable +from checks.syntactic_equivalence import check_mutant_equivalent +from util.io import write_mutant_summary from util.rewrite_method import rewrite_method -def _execute_mutant(project_root): - """Execute the mutated code""" - result = "" - command = f"mvn -f {project_root} clean test" +def _execute(project_root): + """Run the project's tests with Maven. + Returns True if the build passes (all tests green) — i.e. the mutant is not + detected and survives — and False if the build/tests fail (mutant killed). + """ + # argv list (no shell) so a config-controlled project_root cannot inject + # shell commands and paths with spaces are handled correctly. + command = ["mvn", "-f", str(project_root), "clean", "test"] try: - result = subprocess.check_output( - command, shell=True, executable="/bin/bash", stderr=subprocess.STDOUT - ) - + result = subprocess.check_output(command, stderr=subprocess.STDOUT) except subprocess.CalledProcessError as cpe: result = cpe.output + except FileNotFoundError as exc: + raise SystemExit( + "Maven ('mvn') was not found on PATH. Install Maven and a JDK to use " + "the 'mvn' runtool." + ) from exc - finally: - for line in result.splitlines(): - if "BUILD SUCCESS" in line.decode(): - print("Surviving mutant") - break + return b"BUILD SUCCESS" in result def run_mutants( project_root, - filename, + original_file, output_path, method_start_byte, method_end_byte, duplicate, approach, ): - """Execute all mutants using Maven build.""" - mutants_path = Path(output_path, approach + "-mutants") - mutant_files = [ - f for f in os.listdir(mutants_path) if isfile(join(mutants_path, f)) - ] + """Execute all mutants using a Maven build.""" + mutants_dir = Path(output_path, approach + "-mutants") + mutant_files = [f for f in os.listdir(mutants_dir) if isfile(join(mutants_dir, f))] + original_method_path = Path(output_path, "original_method.java") + + mutants = [["MUTANT", "EQUIVALENCE", "COMPILABLE", "SURVIVES"]] + + if not _execute(project_root): + raise IOError( + "The original (unmutated) project did not pass 'mvn clean test', so " + "mutants cannot be evaluated against it. Run it manually to see why: " + f"mvn -f {project_root} clean test" + ) - for file in mutant_files: + for mutant_file in mutant_files: + mutant_output = [mutant_file] + print("Evaluating Mutant: ", mutant_file) if duplicate.exists(): - shutil.copy2(duplicate, filename) + shutil.copy2(duplicate, original_file) + + path = Path(mutants_dir, mutant_file) + mutant_equivalent = check_mutant_equivalent(path, original_method_path) + mutant_output.append(mutant_equivalent) - path = Path(mutants_path, file) - rewrite_method(filename, output_path, method_start_byte, method_end_byte, path) + rewrite_method(original_file, method_start_byte, method_end_byte, path) - mutant_compiles = check_mutant_compilable(filename) + mutant_compiles = check_mutant_compilable(original_file) + mutant_output.append(mutant_compiles) + + mutant_survives = False if mutant_compiles: - check_mutant_compilable(project_root) + mutant_survives = _execute(project_root) + mutant_output.append(str(mutant_survives)) + + mutants.append(mutant_output) - else: - print(f"Mutant {file} does not compile") + write_mutant_summary(mutants_dir, mutants) diff --git a/multiplex/prompts.py b/multiplex/prompts.py new file mode 100644 index 0000000..52daa62 --- /dev/null +++ b/multiplex/prompts.py @@ -0,0 +1,44 @@ +"""Resolve and validate system prompts for the selected mutation approach.""" + +# Each approach uses only its own system prompts. Only the selected approach's +# keys are required for a run; unrelated keys may be omitted from the config. +# Adding an approach means adding its required prompt keys here. +APPROACH_PROMPT_KEYS = { + "basic": ["basic_generate_mutants"], + "hazop": [ + "hazop_describe_process", + "hazop_identify_deviations", + "hazop_implement_deviations", + ], + "stpa": [ + "stpa_describe_control_flow", + "stpa_identify_ucas", + "stpa_implement_ucas", + ], + "mutahunter": ["mutahunter_generate_mutants"], + "llmorpheus": ["llmorpheus_system"], +} + + +def resolve_prompts(config, approach): + """Return the system prompts required by ``approach``. + + Only the keys the selected approach actually uses are read, so a config need + not define prompts for approaches it will not run. Raises ``SystemExit`` with + an actionable message if the approach is unknown or any required + ``system_prompts`` key is missing or empty. + """ + if approach not in APPROACH_PROMPT_KEYS: + valid = ", ".join(sorted(APPROACH_PROMPT_KEYS)) + raise SystemExit(f"Invalid approach '{approach}'. Choose one of: {valid}") + + system_prompts = config.get("system_prompts") or {} + required = APPROACH_PROMPT_KEYS[approach] + missing = [key for key in required if not system_prompts.get(key)] + if missing: + raise SystemExit( + f"Approach '{approach}' requires these system_prompts keys: " + f"{', '.join(missing)}" + ) + + return {key: system_prompts[key] for key in required} diff --git a/tests/execute/conftest.py b/tests/execute/conftest.py new file mode 100644 index 0000000..de5117d --- /dev/null +++ b/tests/execute/conftest.py @@ -0,0 +1,15 @@ +"""Test setup for the execution backends. + +`multiplex/execute/maven.py` (like the other pipeline modules) uses flat imports +—`from checks...`, `from util...`—because at runtime the tool is executed as a +directory with `multiplex/` on `sys.path`. Leaf modules under `multiplex/` +import cleanly as `multiplex.` in tests, but `execute.maven` cannot, so its +tests import it the runtime way. Add `multiplex/` to `sys.path` here. +""" + +import sys +from pathlib import Path + +MULTIPLEX_DIR = Path(__file__).resolve().parents[2] / "multiplex" +if str(MULTIPLEX_DIR) not in sys.path: + sys.path.insert(0, str(MULTIPLEX_DIR)) diff --git a/tests/execute/test_maven.py b/tests/execute/test_maven.py new file mode 100644 index 0000000..31eb1b3 --- /dev/null +++ b/tests/execute/test_maven.py @@ -0,0 +1,183 @@ +import csv +import subprocess +from pathlib import Path + +import pytest + +# `execute.maven` is imported the runtime way (multiplex/ on sys.path); see +# tests/execute/conftest.py. +from execute import maven + +METHOD = "int f(int n) { return n; }" + + +# --------------------------------------------------------------------------- # +# _execute: maps a Maven run to a survives/killed boolean # +# --------------------------------------------------------------------------- # + + +def test_execute_true_on_build_success(monkeypatch): + monkeypatch.setattr( + maven.subprocess, + "check_output", + lambda *a, **k: b"[INFO] Tests run: 3, Failures: 0\n[INFO] BUILD SUCCESS\n", + ) + assert maven._execute("proj") is True + + +def test_execute_false_without_build_success(monkeypatch): + monkeypatch.setattr( + maven.subprocess, + "check_output", + lambda *a, **k: b"[INFO] Tests run: 3, Failures: 1\n[INFO] BUILD FAILURE\n", + ) + assert maven._execute("proj") is False + + +def test_execute_uses_argv_list_not_shell(monkeypatch): + # Guards against shell injection / breakage on paths with spaces: the Maven + # command must be an argv list (no shell), with project_root as one element. + captured = {} + + def fake_check_output(command, *args, **kwargs): + captured["command"] = command + captured["kwargs"] = kwargs + return b"BUILD SUCCESS" + + monkeypatch.setattr(maven.subprocess, "check_output", fake_check_output) + + assert maven._execute("/path with spaces/proj") is True + assert isinstance(captured["command"], list) + assert "/path with spaces/proj" in captured["command"] + assert captured["kwargs"].get("shell", False) is False + + +def test_execute_false_on_called_process_error(monkeypatch): + # A killed mutant makes `mvn test` exit non-zero -> CalledProcessError; the + # captured output is scanned instead of crashing. + def raise_cpe(*a, **k): + raise subprocess.CalledProcessError(1, "mvn", output=b"BUILD FAILURE\n") + + monkeypatch.setattr(maven.subprocess, "check_output", raise_cpe) + assert maven._execute("proj") is False + + +def test_execute_reports_missing_mvn_clearly(monkeypatch): + def raise_fnf(*a, **k): + raise FileNotFoundError(2, "No such file or directory", "mvn") + + monkeypatch.setattr(maven.subprocess, "check_output", raise_fnf) + with pytest.raises(SystemExit) as exc: + maven._execute("proj") + assert "mvn" in str(exc.value).lower() + + +# --------------------------------------------------------------------------- # +# run_mutants: full evaluation loop over a mutants directory # +# --------------------------------------------------------------------------- # + + +@pytest.fixture +def project(tmp_path): + """A synthetic source file + backup, output dir, and three mutants.""" + source = "class C {\n " + METHOD + "\n}\n" + src = tmp_path / "C.java" + src.write_text(source) + (tmp_path / "C.java.orig").write_text(source) + + output = tmp_path / "output" + output.mkdir() + (output / "original_method.java").write_text(METHOD) + + mutants = output / "basic-mutants" + mutants.mkdir() + # equivalent: only a comment differs -> AST-equal, compiles, survives + (mutants / "mutant_equivalent.java").write_text("// same behaviour\n" + METHOD) + # killed: changes behaviour -> compiles, tests fail + (mutants / "mutant_killed.java").write_text("int f(int n) { return n + 1; }") + # broken: unbalanced brace -> does not parse, tests never run + (mutants / "mutant_broken.java").write_text("int f(int n) { { return n; }") + + return { + "project_root": str(tmp_path), + "src": str(src), + "duplicate": Path(str(src) + ".orig"), + "output": output, + "mutants": mutants, + "start": source.index(METHOD), + "end": source.index(METHOD) + len(METHOD), + } + + +def _run(project, approach="basic"): + maven.run_mutants( + project["project_root"], + project["src"], + project["output"], + project["start"], + project["end"], + project["duplicate"], + approach, + ) + + +def _summary(project): + with open(Path(project["mutants"], "mutant_summary.csv")) as f: + rows = list(csv.reader(f)) + return rows[0], {r[0]: r[1:] for r in rows[1:]} + + +def test_run_mutants_writes_summary_with_correct_classification(project, monkeypatch): + src = project["src"] + + # Stand in for `mvn clean test`: the build fails (mutant killed) only when + # the injected source returns n + 1; every other state passes. + monkeypatch.setattr( + maven, "_execute", lambda project_root: "return n + 1" not in Path(src).read_text() + ) + + _run(project) + + header, data = _summary(project) + assert header == ["MUTANT", "EQUIVALENCE", "COMPILABLE", "SURVIVES"] + assert len(data) == 3 + # [EQUIVALENCE, COMPILABLE, SURVIVES] + assert data["mutant_equivalent.java"] == ["True", "True", "True"] + assert data["mutant_killed.java"] == ["False", "True", "False"] + assert data["mutant_broken.java"][1:] == ["False", "False"] # not compilable, killed + + +def test_run_mutants_raises_when_baseline_fails(project, monkeypatch): + monkeypatch.setattr(maven, "_execute", lambda project_root: False) + with pytest.raises(IOError): + _run(project) + + +def test_run_mutants_skips_tests_for_noncompilable_mutant(project, monkeypatch): + # Keep only the non-compilable mutant. + for f in project["mutants"].glob("*.java"): + f.unlink() + (project["mutants"] / "mutant_broken.java").write_text("int f(int n) { { return n; }") + + calls = {"n": 0} + + def counting_execute(project_root): + calls["n"] += 1 + return True + + monkeypatch.setattr(maven, "_execute", counting_execute) + _run(project) + + # Only the baseline runs the suite; the non-compilable mutant is not tested. + assert calls["n"] == 1 + _, data = _summary(project) + assert data["mutant_broken.java"][1:] == ["False", "False"] + + +def test_run_mutants_restores_source_before_each_mutant(project, monkeypatch): + # After the run the backup is untouched and the working file has been left + # holding the last-evaluated mutant (the pipeline's final restore is done by + # __main__.py, not run_mutants). + monkeypatch.setattr(maven, "_execute", lambda project_root: True) + _run(project) + assert (Path(project["duplicate"]).read_text()).count(METHOD) == 1 diff --git a/tests/test_prompts.py b/tests/test_prompts.py new file mode 100644 index 0000000..99ac707 --- /dev/null +++ b/tests/test_prompts.py @@ -0,0 +1,46 @@ +import pytest + +from multiplex.prompts import APPROACH_PROMPT_KEYS, resolve_prompts + + +def _full_config(): + """A config defining every approach's prompts.""" + system_prompts = {} + for keys in APPROACH_PROMPT_KEYS.values(): + for key in keys: + system_prompts[key] = f"prompt for {key}" + return {"system_prompts": system_prompts} + + +@pytest.mark.parametrize("approach", list(APPROACH_PROMPT_KEYS)) +def test_resolve_returns_only_required_subset(approach): + result = resolve_prompts(_full_config(), approach) + assert set(result) == set(APPROACH_PROMPT_KEYS[approach]) + + +def test_resolve_succeeds_with_only_selected_approach_keys(): + # Core of the issue: prompts for other approaches may be absent. + config = {"system_prompts": {"basic_generate_mutants": "do it"}} + assert resolve_prompts(config, "basic") == {"basic_generate_mutants": "do it"} + + +def test_resolve_missing_required_key_raises_naming_it(): + with pytest.raises(SystemExit) as exc: + resolve_prompts({"system_prompts": {}}, "stpa") + assert "stpa_describe_control_flow" in str(exc.value) + + +def test_resolve_empty_required_key_raises(): + with pytest.raises(SystemExit): + resolve_prompts({"system_prompts": {"basic_generate_mutants": ""}}, "basic") + + +def test_resolve_unknown_approach_raises(): + with pytest.raises(SystemExit) as exc: + resolve_prompts(_full_config(), "nope") + assert "Invalid approach" in str(exc.value) + + +def test_resolve_missing_system_prompts_section_raises(): + with pytest.raises(SystemExit): + resolve_prompts({}, "basic")