Skip to content

About

Fail CI when quantization costs you task accuracy. Task-level regression testing for quantized models on CPU.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

quant-regress

Fail CI when quantization costs you task accuracy.

Token-level proxies — perplexity, KL divergence — routinely report that a quantized model is fine. Task accuracy does not, provided you actually measure it. This action measures a task eval at each precision and fails the build when accuracy drops past a threshold you set.

If you pinned @v1 before 2026-10-15, re-sync. The tag has been re-pointed at least four times, most recently on 2026-10-15, each time to pick up correctness fixes — including one where the gate reported PASS for a model it had never actually quantized. For a gate you rely on, pin a SHA rather than a moving tag. See CHANGELOG.md.

Measured on a real checkpoint

The headline is a real measurement, not a stub. scripts/regen_real_model.py loads a real instruction-tuned causal LM — Qwen/Qwen2.5-0.5B-Instruct — and runs 20 single-answer arithmetic word problems through the same QuantHarness and adapter path the GitHub Action uses: an fp32 baseline and a dynamic-int8 candidate, scored on the numeral the model produces. The fp32 baseline clears the --min-accuracy floor, so the int8 delta below is a real comparison, not a measurement from a model that cannot do the task.

precision accuracy correct seconds
fp32 (baseline) 80.0% 16/20 81.4s
int8 0.0% 0/20 154.2s

worst drop: +80.0 pts -> FAIL

baseline 80.0% is above the --min-accuracy floor (0.5)

That is the failure this action exists to catch, on a real checkpoint: the quantized model still runs and still returns text, but the text is wrong. The int8 completions degrade into conversational filler ("To find the number that is 7", "I'm here to answer any question you") instead of the answer, so a token-level proxy that only looks at the distribution can clear it while the task eval — which asks for the numeral — cannot. Regenerate with python scripts/regen_real_model.py --measure (CPU, several minutes); the committed run is at results/real_model.json, and tests/test_claims_are_reproducible.py fails the build if a number in this table stops matching the script's output.

The controlled reproduction (stub)

The original motivation was a real cross-encoder claim verifier in citesure, where dynamic INT8 left aggregate accuracy looking survivable while destroying the capability that mattered. This repo's adapter loads AutoModelForCausalLM, so it cannot load a cross-encoder at all — the table below is therefore this repo's controlled reproduction of that shape, not a run of that model: a stub whose only behaviour is to separate at fp32 and collapse at int8, so the gate has something deterministic to catch in its own CI. Read its figures as a test of the gate's arithmetic, not as a model measurement.

precision accuracy correct
fp32 (baseline) 100.0% 20/20
int8 0.0% 0/20

Regenerate with python scripts/regen_int8_collapse.py; the run's output is committed at results/int8_collapse.json.

The pattern that matters in both tables: the quantized model still runs, still returns a valid answer, and still looks like a working model. It is simply wrong every time. That is why a token-level proxy can clear it and a task eval cannot.

Usage

As a GitHub Action:

- uses: DawnofGenX/quant-regress@v1
  with:
    eval: evals/tasks.jsonl
    model: meta-llama/Llama-3.2-1B-Instruct
    max-drop-points: 2
    min-accuracy: 0.5      # optional; see "Change is not quality" below

@v1 is a moving tag — it tracks the current release line, and has been re-pointed at least four times (to pick up the fixes in CHANGELOG.md). Pin a commit SHA if you need immutability.

Or from the command line / your own CI:

pip install quant-regress

quant-regress --eval evals/tasks.jsonl \
              --model meta-llama/Llama-3.2-1B-Instruct \
              --max-drop-points 2 \
              --min-accuracy 0.5 \
              --report quant-regress-report.json

Exit codes: 0 within tolerance, 1 accuracy regressed, 2 bad configuration.

--dry-run checks the whole configuration and exits without downloading a model — a typo'd precision, a baseline equal to a candidate, or a broken eval set costs milliseconds instead of a multi-GB download. --version prints the installed version, for pinning a ref or filing a bug.

Outputs

--report writes JSON containing every arm's accuracy, the drop in points, and the misclassified case ids — so a red build names the cases that got worse instead of only saying that something did. --output-format junit --junit-path results.xml additionally writes JUnit XML with one <testcase> per precision, each failing arm carrying a <failure> naming the drop and the cases. That is what Jenkins, GitLab and the GitHub test-results pane render.

--progress-every 25 prints a progress line to stderr every N cases. Stdout stays a parseable table, so a workflow that greps the summary line is unaffected.

Matching a chatty model

--scorer contains matches when expected appears anywhere in the prediction, for a model that answers in a sentence. The default is exact — normalised equality, which is what makes a marginal-decision drop visible in the first place. Only those two are exposed on the CLI; a scorer that needs real logic belongs in the library, via QuantHarness(..., scorer=fn). Deliberately not an arbitrary expression: this tool runs on arbitrary PRs, and evaluating one would be code execution.

--labels "no,yes" names the classes for a logits-only model with no id2label. --system-prompt prepends a prompt to every case. --hf-token (or $HF_TOKEN) authenticates a gated model.

Chat models

An instruction-tuned checkpoint is prompted through its own chat template, not by pasting text. A base model's tokenizer has no template, so it keeps the plain concatenation — --system-prompt behaves as before. This matters because a chat model fed a hand-pasted prompt answers conversationally rather than with the expected token, so the accuracy being measured would be the accuracy of a broken prompt, and the marginal-decision damage this tool exists to detect would be invisible.

--min-accuracy is worth setting. Without it the gate only measures change, so a model that is wrong at every precision yields a zero-point drop and passes — a green build for a model that cannot do the task at all. --min-accuracy is a fraction (0.5 = half) and fails the build when the baseline is that weak. It is a quality failure, so it exits 1, not 2: the tool measured correctly and the answer is no. See change is not quality.

Constraints worth knowing

  • CPU only. quantize_dynamic emits CPU-only quantized kernels; moving the model to CUDA raises at forward time. GitHub-hosted runners have no GPU, so this is not a limitation in CI — but it does mean GPU-only quantizers (AWQ/GPTQ/GGUF) are out of scope for v1.
  • Not every architecture can be quantized by this method, and the action refuses rather than lying. quantize_dynamic only replaces module types in torch's own mapping table, which covers nn.Linear. Architectures whose blocks use something else — GPT-2 and GPT-J use transformers.Conv1D — would have their transformer body left in fp32, so the "int8" arm would really be fp32 and the tool would report a 0.00-point drop and PASS for a model it never quantized. That architecture now exits 2 (misconfigured) with an actionable message instead. Use an nn.Linear-based architecture: BERT, DeBERTa, Llama, Mistral.
  • Only the model's continuation is scored. generate() returns the prompt followed by the new tokens, so the prediction is what the model added — not the question you asked it. Prompts should ask for a short, exact answer, and expected should be that answer. A chatty model will not exact-match: use --scorer contains, or pass a scorer via QuantHarness(..., scorer=fn) when using the library.
  • The threshold unit is accuracy points, not a ratio. --max-drop-points: 2 means "fail if any candidate is more than 2 points below the baseline".
  • The two gates use different units, deliberately. --max-drop-points is in percentage points; --min-accuracy is a fraction — 0.5, not 50. A percentage-style value is rejected with an explanatory error rather than silently clamped, because a typo there would disable the floor instead of tightening it.
  • Comparison is exact-after-normalisation by default. Pass a scorer if you need semantic equivalence.
  • Models are downloaded at run time. A 1B model is ~2 GB; check runner disk.

Cost and caching

A run downloads the model once (HF cache) and then generates on every case at every precision, on CPU, one case at a time. A 20-case, two-precision eval on a 1B model is minutes per invocation, and every PR re-pays it. Two levers:

  • --response-cache DIR caches each prediction keyed by (model, precision, prompt). A re-run of an unchanged eval against an unchanged model skips generation entirely and finishes in seconds; the cache is a single responses.json you can commit or share between runners. When every case is cached, the model is not loaded at all, so a fully cached run never touches the network or the weights.
  • Batching across cases is not implemented yet: prompts go through the model's own chat template with variable length, and naive padding changes which tokens are scored. Until that is solved honestly, generation stays per-case.

If neither lever applies, budget the run accordingly and read the baseline accuracy in the report before trusting a PASS.

Honest limits

  • Only fp32, fp16, bf16 and dynamic int8 are supported in v1. AWQ / GPTQ / GGUF / NF4 need external tooling; the adapter is a registry (src/quant_regress/adapter.py, BACKENDS) so a new backend is a one-function addition, but each must ship with a committed measurement before it is advertised.
  • The library runs generation, so it needs a task whose success is checkable in code. It is not an LLM-judge harness.
  • The bundled evals/fixtures/collapse.jsonl is model-specific, and it does not exercise the generate path. Its cases expect the answer yes, so it only demonstrates a collapse for a model that answers yes when unquantized. Pointed at an untuned tiny LM it scores 0/N at both precisions, produces no drop, and the gate correctly reports PASS. It is also driven by a stub model exposing .answer(), so it tells you nothing about real decoding. Use it to see the output format; bring your own eval for a real check.

Change is not quality — read this before trusting a PASS

The gate's default judgement is about change: it fails when quantization costs accuracy. That means a model that is equally bad at both precisions produces a zero-point drop and passes. Measured on a ten-case set: a model answering nothing correctly at either precision reports worst drop: +0.00 pts -> PASS and exit 0 , and the same model with --min-accuracy 0.5 reports FAIL. Regenerate with python scripts/regen_quiet_model.py; the run's output is committed at results/quiet_model.json.

That is correct for the tool's purpose — quantization did not cause it — but it means a green build is not evidence that the model works. Two options:

  • Pass --min-accuracy 0.5 (or your own floor) so the baseline is gated too. This turns the case above into FAIL / exit 1, with the reason printed.
  • Read the baseline accuracy in the report (results/*.json, or the baseline accuracy: line) before trusting a PASS. The number has always been recorded; nothing enforced it.

A floor breach is reported as a quality failure (exit 1), not a configuration error (exit 2), because the tool measured correctly and the answer is simply no.

  • What the repo's own CI pins is the exit-code contract (0 holds, 1 regressed, 2 misconfigured), not any particular model's accuracy — see tests/test_selftest_gate.py.
  • No published adoption yet. If you use it, an issue saying what you quantized and what broke is genuinely useful.

More writing on this stack: https://dev.to/dawnofgenx

About

Fail CI when quantization costs you task accuracy. Task-level regression testing for quantized models on CPU.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages