Fail CI when quantization costs you task accuracy.
Token-level proxies — perplexity, KL divergence — routinely report that a quantized model is fine. Task accuracy does not, provided you actually measure it. This action measures a task eval at each precision and fails the build when accuracy drops past a threshold you set.
If you pinned
@v1before 2026-10-15, re-sync. The tag has been re-pointed at least four times, most recently on 2026-10-15, each time to pick up correctness fixes — including one where the gate reported PASS for a model it had never actually quantized. For a gate you rely on, pin a SHA rather than a moving tag. SeeCHANGELOG.md.
The headline is a real measurement, not a stub. scripts/regen_real_model.py
loads a real instruction-tuned causal LM — Qwen/Qwen2.5-0.5B-Instruct — and
runs 20 single-answer arithmetic word problems through the same QuantHarness
and adapter path the GitHub Action uses: an fp32 baseline and a dynamic-int8
candidate, scored on the numeral the model produces. The fp32 baseline clears
the --min-accuracy floor, so the int8 delta below is a real comparison, not a
measurement from a model that cannot do the task.
| precision | accuracy | correct | seconds |
|---|---|---|---|
| fp32 (baseline) | 80.0% | 16/20 | 81.4s |
| int8 | 0.0% | 0/20 | 154.2s |
worst drop: +80.0 pts -> FAIL
baseline 80.0% is above the --min-accuracy floor (0.5)
That is the failure this action exists to catch, on a real checkpoint: the
quantized model still runs and still returns text, but the text is wrong. The
int8 completions degrade into conversational filler ("To find the number that
is 7", "I'm here to answer any question you") instead of the answer, so a
token-level proxy that only looks at the distribution can clear it while the
task eval — which asks for the numeral — cannot. Regenerate with
python scripts/regen_real_model.py --measure (CPU, several minutes); the
committed run is at results/real_model.json, and
tests/test_claims_are_reproducible.py fails the build if a number in this
table stops matching the script's output.
The original motivation was a real cross-encoder claim verifier in
citesure, where dynamic INT8 left
aggregate accuracy looking survivable while destroying the capability that
mattered. This repo's adapter loads AutoModelForCausalLM, so it cannot load
a cross-encoder at all — the table below is therefore this repo's controlled
reproduction of that shape, not a run of that model: a stub whose only
behaviour is to separate at fp32 and collapse at int8, so the gate has something
deterministic to catch in its own CI. Read its figures as a test of the gate's
arithmetic, not as a model measurement.
| precision | accuracy | correct |
|---|---|---|
| fp32 (baseline) | 100.0% | 20/20 |
| int8 | 0.0% | 0/20 |
Regenerate with python scripts/regen_int8_collapse.py; the run's output is
committed at results/int8_collapse.json.
The pattern that matters in both tables: the quantized model still runs, still returns a valid answer, and still looks like a working model. It is simply wrong every time. That is why a token-level proxy can clear it and a task eval cannot.
As a GitHub Action:
- uses: DawnofGenX/quant-regress@v1
with:
eval: evals/tasks.jsonl
model: meta-llama/Llama-3.2-1B-Instruct
max-drop-points: 2
min-accuracy: 0.5 # optional; see "Change is not quality" below@v1 is a moving tag — it tracks the current release line, and has been re-pointed at least four
times (to pick up the fixes in CHANGELOG.md). Pin a commit SHA if you need immutability.
Or from the command line / your own CI:
pip install quant-regress
quant-regress --eval evals/tasks.jsonl \
--model meta-llama/Llama-3.2-1B-Instruct \
--max-drop-points 2 \
--min-accuracy 0.5 \
--report quant-regress-report.jsonExit codes: 0 within tolerance, 1 accuracy regressed, 2 bad configuration.
--dry-run checks the whole configuration and exits without downloading a
model — a typo'd precision, a baseline equal to a candidate, or a broken eval
set costs milliseconds instead of a multi-GB download. --version prints the
installed version, for pinning a ref or filing a bug.
--report writes JSON containing every arm's accuracy, the drop in points, and the
misclassified case ids — so a red build names the cases that got worse instead of
only saying that something did. --output-format junit --junit-path results.xml
additionally writes JUnit XML with one <testcase> per precision, each failing arm
carrying a <failure> naming the drop and the cases. That is what Jenkins, GitLab
and the GitHub test-results pane render.
--progress-every 25 prints a progress line to stderr every N cases. Stdout
stays a parseable table, so a workflow that greps the summary line is unaffected.
--scorer contains matches when expected appears anywhere in the prediction, for
a model that answers in a sentence. The default is exact — normalised equality,
which is what makes a marginal-decision drop visible in the first place. Only those
two are exposed on the CLI; a scorer that needs real logic belongs in the library,
via QuantHarness(..., scorer=fn). Deliberately not an arbitrary expression: this
tool runs on arbitrary PRs, and evaluating one would be code execution.
--labels "no,yes" names the classes for a logits-only model with no id2label.
--system-prompt prepends a prompt to every case. --hf-token (or $HF_TOKEN)
authenticates a gated model.
An instruction-tuned checkpoint is prompted through its own chat template, not
by pasting text. A base model's tokenizer has no template, so it keeps the plain
concatenation — --system-prompt behaves as before. This matters because a chat
model fed a hand-pasted prompt answers conversationally rather than with the
expected token, so the accuracy being measured would be the accuracy of a broken
prompt, and the marginal-decision damage this tool exists to detect would be
invisible.
--min-accuracy is worth setting. Without it the gate only measures change, so a
model that is wrong at every precision yields a zero-point drop and passes — a green build
for a model that cannot do the task at all. --min-accuracy is a fraction (0.5 = half) and
fails the build when the baseline is that weak. It is a quality failure, so it exits 1,
not 2: the tool measured correctly and the answer is no. See
change is not quality.
- CPU only.
quantize_dynamicemits CPU-only quantized kernels; moving the model to CUDA raises at forward time. GitHub-hosted runners have no GPU, so this is not a limitation in CI — but it does mean GPU-only quantizers (AWQ/GPTQ/GGUF) are out of scope for v1. - Not every architecture can be quantized by this method, and the action
refuses rather than lying.
quantize_dynamiconly replaces module types in torch's own mapping table, which coversnn.Linear. Architectures whose blocks use something else — GPT-2 and GPT-J usetransformers.Conv1D— would have their transformer body left in fp32, so the "int8" arm would really be fp32 and the tool would report a 0.00-point drop and PASS for a model it never quantized. That architecture now exits2(misconfigured) with an actionable message instead. Use annn.Linear-based architecture: BERT, DeBERTa, Llama, Mistral. - Only the model's continuation is scored.
generate()returns the prompt followed by the new tokens, so the prediction is what the model added — not the question you asked it. Prompts should ask for a short, exact answer, andexpectedshould be that answer. A chatty model will not exact-match: use--scorer contains, or pass a scorer viaQuantHarness(..., scorer=fn)when using the library. - The threshold unit is accuracy points, not a ratio.
--max-drop-points: 2means "fail if any candidate is more than 2 points below the baseline". - The two gates use different units, deliberately.
--max-drop-pointsis in percentage points;--min-accuracyis a fraction —0.5, not50. A percentage-style value is rejected with an explanatory error rather than silently clamped, because a typo there would disable the floor instead of tightening it. - Comparison is exact-after-normalisation by default. Pass a scorer if you need semantic equivalence.
- Models are downloaded at run time. A 1B model is ~2 GB; check runner disk.
A run downloads the model once (HF cache) and then generates on every case at every precision, on CPU, one case at a time. A 20-case, two-precision eval on a 1B model is minutes per invocation, and every PR re-pays it. Two levers:
--response-cache DIRcaches each prediction keyed by(model, precision, prompt). A re-run of an unchanged eval against an unchanged model skips generation entirely and finishes in seconds; the cache is a singleresponses.jsonyou can commit or share between runners. When every case is cached, the model is not loaded at all, so a fully cached run never touches the network or the weights.- Batching across cases is not implemented yet: prompts go through the model's own chat template with variable length, and naive padding changes which tokens are scored. Until that is solved honestly, generation stays per-case.
If neither lever applies, budget the run accordingly and read the baseline accuracy in the report before trusting a PASS.
- Only
fp32,fp16,bf16and dynamicint8are supported in v1. AWQ / GPTQ / GGUF / NF4 need external tooling; the adapter is a registry (src/quant_regress/adapter.py,BACKENDS) so a new backend is a one-function addition, but each must ship with a committed measurement before it is advertised. - The library runs generation, so it needs a task whose success is checkable in code. It is not an LLM-judge harness.
- The bundled
evals/fixtures/collapse.jsonlis model-specific, and it does not exercise the generate path. Its cases expect the answeryes, so it only demonstrates a collapse for a model that answersyeswhen unquantized. Pointed at an untuned tiny LM it scores 0/N at both precisions, produces no drop, and the gate correctly reports PASS. It is also driven by a stub model exposing.answer(), so it tells you nothing about real decoding. Use it to see the output format; bring your own eval for a real check.
The gate's default judgement is about change: it fails when quantization costs
accuracy. That means a model that is equally bad at both precisions produces a
zero-point drop and passes. Measured on a ten-case set:
a model answering nothing correctly at either precision reports
worst drop: +0.00 pts -> PASS and exit 0 , and the same
model with --min-accuracy 0.5 reports FAIL. Regenerate with
python scripts/regen_quiet_model.py; the run's output is committed at
results/quiet_model.json.
That is correct for the tool's purpose — quantization did not cause it — but it means a green build is not evidence that the model works. Two options:
- Pass
--min-accuracy 0.5(or your own floor) so the baseline is gated too. This turns the case above intoFAIL/ exit1, with the reason printed. - Read the baseline accuracy in the report (
results/*.json, or thebaseline accuracy:line) before trusting a PASS. The number has always been recorded; nothing enforced it.
A floor breach is reported as a quality failure (exit 1), not a configuration
error (exit 2), because the tool measured correctly and the answer is simply no.
- What the repo's own CI pins is the exit-code contract (0 holds, 1 regressed,
2 misconfigured), not any particular model's accuracy — see
tests/test_selftest_gate.py. - No published adoption yet. If you use it, an issue saying what you quantized and what broke is genuinely useful.
More writing on this stack: https://dev.to/dawnofgenx