data: FP benchmark + gap-fix training set for transactional A2P by Frhnfaya · Pull Request #209 · TelecomsXChangeAPi/OpenTextShield

Frhnfaya · 2026-06-26T10:31:54Z

Follows up on the false-positive discussion in #202.

What this adds

ham_benchmark_v1.csv — 88-msg held-out benchmark (en/es/it/pt, 8 categories). Block rate on this set = false-positive rate. Locked — never train on it.
ham_training_v1.csv — 486-row training set: 282 realistic branded A2P ham + 204 hard-negative scam twins (labelled phishing). Gap-weighted toward government and delivery where FP is worst. Zero overlap with benchmark.
build_benchmark.py / build_training.py — reproducible generators.
README_fp_fix.md — problem, root cause, baseline numbers, usage.
evals/results/summary_baseline.json — model 2.7 baseline evidence.

Baseline (model 2.7, 88-msg benchmark)

overall FP 67% (59/88), all → spam, none → phishing
government 94%, delivery 92%, appointment/telecom 75%, bank_alert 67%
personal/personal_money 0% — conversational ham is clean

Reproduction

python src/mBERT/tests/hamset/build_benchmark.py
python evals/run_eval.py \
  --model src/mBERT/training/model-training/mbert_ots_model_2.7.pth \
  --dataset csv:src/mBERT/tests/hamset/ham_benchmark_v1.csv \
  --tag baseline

Expected: overall FP 67%, government 94%, delivery 92%.

Honest caveat: both sets are constructed-realistic, not scraped. The before/after proves the model can learn the A2P-is-ham boundary; real operator A2P would be the gold-standard confirmation on top.

…rovement Evaluate the OTS mBERT classifier against public SMS-classification benchmarks and a new multilingual adversarial suite, then close the identified gaps with a targeted synthetic dataset and an incremental fine-tune. Findings (v2.5 baseline): - 99.3% accuracy on the UCI SMS Spam Collection (classic benchmark) - 99.2% block rate on the Mishra & Soni SMS Phishing dataset, but only 2.7% phishing-label recall (smishing routed to spam) - 20.4% three-class / 69.9% block rate on the modern, multilingual IMC 2025 smishing corpus, with ~25% of phishing silently passed as ham - Non-English block rates falling to ~52-56% Improvement (v2.6 incremental fine-tune): - UCI block rate held at 99.2% (no classic-spam regression) - IMC25 block rate 69.9% -> 78.2%, phishing recall 17% -> 40% - Adversarial suite block rate 73.3% -> 98.0%, phishing recall 11.5% -> 82% - Multilingual block-rate gains concentrated where the model was weak (Italian +19.6pts, Japanese +19.7pts, Spanish +13.8pts) Adds an offline, reproducible evaluation harness, the adversarial test set, the Mishra & Soni benchmark loader, a synthetic data generator with locale-correct brands and hard-negative ham twins, an incremental fine-tune script, and a full written report (evals/REPORT.md).

…ardening Training (evals/finetune_tier1.py): - Apple Silicon MPS / CUDA / CPU device auto-detection - stratified train/validation split with per-epoch validation metrics - best-checkpoint selection by validation macro-F1 (fixes the last-epoch regressions from the proof-of-concept run) - class-weighted or focal loss to counter the phishing->spam collapse - warmup + linear LR schedule, gradient clipping Calibration (evals/calibrate_thresholds.py + run_eval --logit-bias): - training-free per-class logit-bias search to recover phishing recall; on the adversarial set this alone lifts block 73%->83% and phishing recall 12%->35% on the unchanged v2.5 weights Obfuscation hardening (production path): - strip zero-width / invisible formatting characters in the preprocessor - run homoglyph + zero-width normalisation in the dynamic-batching path (previously only the per-request path normalised, so the production default never did) Adds evals/TIER1.md runbook for reproducing on Apple Silicon.

- ModelInfo gains an 'architecture' field so 'version' is unambiguously the model semver (e.g. 2.7), not the network name; fix misleading schema examples - /health now reports both api_version (platform, e.g. 2.10.0) and a model block (name/version/architecture); 'version' kept as a backward-compatible alias - add model_manager.get_model_version() for a cheap version lookup

…obustness, normalisation tests - Untrack reproducible prediction dumps; gitignore catch-all (review TelecomsXChangeAPi#3) - Vendored BERT vocab -> Git LFS + Apache-2.0 attribution (review TelecomsXChangeAPi#1) - calibrate_thresholds: importlib load (no sys.path mutation) + with-statement (review TelecomsXChangeAPi#4,TelecomsXChangeAPi#5) - run_eval: ap.error instead of assert; DEFAULT_MODEL baseline note (review TelecomsXChangeAPi#8,TelecomsXChangeAPi#9) - finetune_incremental: --num-workers arg, default 0 (macOS safety) (review TelecomsXChangeAPi#7) - settings: OTS_MBERT_MODEL_PATH override for env-driven model rollback (review TelecomsXChangeAPi#2) - health/response_models: clarify architecture/tokenizer key; mark version deprecated (review TelecomsXChangeAPi#10,TelecomsXChangeAPi#11) - add 9 unit tests for the batching-path normalisation + fallback (review TelecomsXChangeAPi#6)

…ers module, CI, perf, tests Bugs: - settings: guard _apply_model_path_override against unknown default_mbert_version (KeyError) (TelecomsXChangeAPi#1) - run_eval: warn when scoring the v2.5 baseline default instead of production weights (TelecomsXChangeAPi#3) Quality: - batching_service: log a warning when EnhancedPreprocessor import fails (no silent disable) (TelecomsXChangeAPi#4) - extract shared dataset loaders to evals/loaders.py; run_eval + calibrate import it normally (drops importlib hack) (TelecomsXChangeAPi#5) - drop unreferenced intermediate summaries summary_v2.5_mishra/v2.6/v2.6_big.json (TelecomsXChangeAPi#6) - health: de-rot the architecture/tokenizer-key comment (no review-number ref) (TelecomsXChangeAPi#9) - enhanced_preprocessing: INVISIBLE_CHARS as \u escapes with codepoint comments (set unchanged) (minor) Performance: - batching_service: ASCII fast-path in _normalize_text skips normalisation for typical traffic (TelecomsXChangeAPi#7) API: - response_models: mark HealthResponse.version Field(deprecated=True) (TelecomsXChangeAPi#8) Tests/CI: - add GitHub Actions unit-test workflow running pytest (TelecomsXChangeAPi#12) - add settings-override tests incl. KeyError guard + per-instance isolation (TelecomsXChangeAPi#13) - add ASCII fast-path test; keep raise-swallow test on non-ASCII input (TelecomsXChangeAPi#7) Docs: - TIER1.md: loosen torch pin to stable; flag in-domain validation caveat (minor) - dataset/README_SYNTHETIC.md: mark synthetic data, warn against pipeline misuse (TelecomsXChangeAPi#11)

Addresses CodeQL actions hardening finding: limit GITHUB_TOKEN scope.

- load_imc25: skip and count rows with no scam_type instead of silently labelling them phishing; document IMC25 as an all-attack corpus (no ham class), confirming the spam/phishing mapping is correct for it - use md5(usedforsecurity=False) for the deterministic placeholder seed so the loader works under FIPS-enforced environments - REPORT.md: note IMC25 measures attack recall only (no false-positive signal), flag the missing multilingual legitimate-message control as the top next addition, and mark the batching-path obfuscation normalization as done rather than pending

- ham_benchmark_v1.csv: 88-msg held-out benchmark (en/es/it/pt, 8 categories) - ham_training_v1.csv: 486-row training set (282 ham + 204 phishing hard-negatives) - build_benchmark.py / build_training.py: reproducible generators - README_fp_fix.md: problem, baseline numbers, usage - summary_baseline.json: model 2.7 baseline (67% FP overall, government 94%) Zero overlap between benchmark and training set (leakage-free). Closes the gap identified in TelecomsXChangeAPi#202 comments: transactional A2P mislabelled spam.

ajamous · 2026-07-02T00:21:53Z

@Frhnfaya thanks for putting this together — the FP gap is real and this is exactly the control set the eval was missing. Two things I need resolved before merging:

1. The baseline doesn't reproduce. Running the PR's own repro command (evals/run_eval.py --model mbert_ots_model_2.7.pth --dataset csv:ham_benchmark_v1.csv) against the released 2.7 weights (v2.10.0-rc.1 asset) I get 32/88 FP = 36.4%, 13→spam / 19→phishing — not 59/88 = 67% all-spam. Per-category differs too: delivery 33% vs your 92%, appointment 12.5% vs 75%, otp 0% vs 50% (government 87.5% is the only one close). Which weights + command did you run for summary_baseline.json? If it was a different checkpoint, let's regenerate the committed baseline with run_eval.py against the released 2.7 — and the WEIGHT table in build_training.py should follow the corrected numbers (government looks like the real worst category, not delivery).

2. The leakage guard is exact-string only, and the two sets share templates. 21/88 benchmark rows have a ≥0.85 difflib match in ham_training_v1.csv (7 at ≥0.95, e.g. the TIM/MyTIM pair at 0.989, the EE bill £28.50 vs £42.50 at 0.988), concentrated in delivery/telecom/bank_alert. Fine-tuning on those and measuring on the "locked" benchmark would partly measure memorization. Can you rewrite the overlapping benchmark rows with distinct phrasings (or dedup at a similarity threshold) so the leakage-free claim holds?

Even at 36%, the transactional-A2P problem stands — government at 87.5% is genuinely broken — so I want this in as the FP gate before 2.7 graduates from RC. Happy to merge once the baseline reproduces and the templates are disjoint.

Minor: README references summarize.py which isn't in the PR.

ajamous · 2026-07-02T01:47:28Z

Heads-up: #202 has merged, so this PR is now retargeted to main. GitHub reports conflicts because the branch predates the eval-suite decontamination that landed on the eval branch after you forked it — nothing in your six files actually collides.

Easiest fix, since this PR is purely additive: start a fresh branch from current main, copy your six files over, and force-push (or rebase and take main's side for evals/REPORT.md, evals/datasets/fable5_adversarial_v1.csv, evals/results/summary_v2.5.json, summary_v2.7.json, and evals/assets/ATTRIBUTION.md).

Everything you need is now on main: evals/run_eval.py, the model 2.7 weights, and the decontaminated adversarial suite. The two asks from my review comment above (regenerate summary_baseline.json against the released 2.7 via run_eval.py, and rewrite the ~21 benchmark templates that near-duplicate the training rows) still stand — once those are in, this merges as the FP gate for the 2.8 work.

Frhnfaya · 2026-07-03T19:24:53Z

Heads-up: #202 has merged, so this PR is now retargeted to main. GitHub reports conflicts because the branch predates the eval-suite decontamination that landed on the eval branch after you forked it — nothing in your six files actually collides.

Easiest fix, since this PR is purely additive: start a fresh branch from current main, copy your six files over, and force-push (or rebase and take main's side for evals/REPORT.md, evals/datasets/fable5_adversarial_v1.csv, evals/results/summary_v2.5.json, summary_v2.7.json, and evals/assets/ATTRIBUTION.md).

Everything you need is now on main: evals/run_eval.py, the model 2.7 weights, and the decontaminated adversarial suite. The two asks from my review comment above (regenerate summary_baseline.json against the released 2.7 via run_eval.py, and rewrite the ~21 benchmark templates that near-duplicate the training rows) still stand — once those are in, this merges as the FP gate for the 2.8 work.

@ajamous Dug into this — it's not a checkpoint mismatch, it's a calibration mismatch. run_eval.py
has no default bias; results depend entirely on --logit-bias, which my repro command
didn't include.

Tried two settings against the released 2.7 weights on the benchmark:

uncalibrated: 59/88 FP (67%), all→spam, none→phishing — matches my original numbers
calibrated via calibrate_thresholds.py's documented Mishra example
(bias 0.0,-2.0,0.5): 11/88 FP (12.5%), government 25% FP — neither matches your
32/88 (36.4%) with a 13-spam/19-phishing split and government at 87.5%.

Could you share the exact --logit-bias value (or command) you used for the release
comparison? Once I have it I'll add it to the repro steps and regenerate
summary_baseline.json against it so the numbers match exactly.

Separately — even at your calibrated 36.4%, government is still the worst category at
87.5% FP, so the core transactional-A2P finding holds regardless of which bias setting
we standardize on.

ajamous · 2026-07-03T19:36:23Z

@Frhnfaya No bias at all. The 36.4% run is uncalibrated. Without the --logit-bias flag the harness just takes the raw argmax, so calibration can't explain the difference between our runs.

I re-ran it today on your exact benchmark file from this PR:

python evals/run_eval.py \
    --model src/mBERT/training/model-training/mbert_ots_model_2.7.pth \
    --dataset csv:ham_benchmark_v1.csv

Result: 32/88 FP (36.4%). 13 went to spam, 19 to phishing. Worst categories: government 87.5% FP, telecom 75%, delivery 33%. Env: torch 2.8.0, transformers 4.53.0, CPU.

Same command, same code. So the only thing left that can differ is the weights file. Mine:

SHA-256: 52fe17755fd3ecd3609b19199c895e8eeb126a0b527791de52aee00c7f2efd36
size:    711,504,880 bytes (matches the v2.10.0-rc.1 release asset)

Can you run shasum -a 256 on your copy? A run that predicts zero phishing on 88 messages doesn't look like the released 2.7. That model leans heavily toward phishing predictions. My guess is you re-trained 2.7 locally with finetune_tier1.py, and a local retrain will not match the released file.

Agreed on the main point: government is the worst category in both of our runs, so the A2P problem is real either way. Once we confirm the weights match, let's use uncalibrated + released 2.7 as the standard baseline, regenerate summary_baseline.json with the command above, and update the WEIGHT table to match. Then this is good to merge along with the template dedup fix.

Frhnfaya · 2026-07-04T15:21:49Z

@Frhnfaya No bias at all. The 36.4% run is uncalibrated. Without the --logit-bias flag the harness just takes the raw argmax, so calibration can't explain the difference between our runs.

I re-ran it today on your exact benchmark file from this PR:
python evals/run_eval.py \
    --model src/mBERT/training/model-training/mbert_ots_model_2.7.pth \
    --dataset csv:ham_benchmark_v1.csv
Result: 32/88 FP (36.4%). 13 went to spam, 19 to phishing. Worst categories: government 87.5% FP, telecom 75%, delivery 33%. Env: torch 2.8.0, transformers 4.53.0, CPU.

Same command, same code. So the only thing left that can differ is the weights file. Mine:
SHA-256: 52fe17755fd3ecd3609b19199c895e8eeb126a0b527791de52aee00c7f2efd36
size:    711,504,880 bytes (matches the v2.10.0-rc.1 release asset)
Can you run shasum -a 256 on your copy? A run that predicts zero phishing on 88 messages doesn't look like the released 2.7. That model leans heavily toward phishing predictions. My guess is you re-trained 2.7 locally with finetune_tier1.py, and a local retrain will not match the released file.

Agreed on the main point: government is the worst category in both of our runs, so the A2P problem is real either way. Once we confirm the weights match, let's use uncalibrated + released 2.7 as the standard baseline, regenerate summary_baseline.json with the command above, and update the WEIGHT table to match. Then this is good to merge along with the template dedup fix.

@ajamous finally Confirmed — weights match (same SHA-256), and evals/run_eval.py + loaders.py are
identical to main (no diff). So it's environment: I'm on torch 2.10.0+cpu /
transformers 5.12.1, you're on torch 2.8.0 / transformers 4.53.0. Given identical
weights + code, that's the only variable left, and it fits — later transformers/torch
versions can shift borderline logits enough to flip predictions right at the decision
boundary, which is exactly where an uncalibrated 3-class head lives.

I'll pin to torch==2.8.0 / transformers==4.53.0 in a venv, rerun, and confirm I get
your 32/88 (36.4%) split. If it matches, I'll regenerate summary_baseline.json under
those pinned versions and note the exact env in the PR so it's reproducible going
forward — might also be worth a requirements-pin for evals/ generally, since this bit
us here.

Government being the worst category either way is good confirmation the core finding
is robust to the version difference.

Frhnfaya · 2026-07-04T15:31:50Z

@ajamous Traced it further — I can't even install torch==2.8.0 in a clean venv; PyPI has no
wheel for it on my Python (3.14), only 2.9.0+. So the version gap isn't something I
can easily replicate locally without also downgrading Python.

Rather than chase exact version parity, could you try one thing on your end: rerun
against a newer torch/transformers (e.g. whatever "pip install -U torch transformers"
gives you) and see if you get my 59/88 uncalibrated result? That would confirm the
library-version sensitivity directly, and we can decide from there whether to pin
evals/ to your versions in requirements or just document both results.

Either way — happy to standardize on whichever baseline you want as the source of
truth for summary_baseline.json; just let me know which env to target and I'll match
it as closely as my setup allows.

ajamous and others added 10 commits June 13, 2026 02:02

Release model 2.7: validated fine-tune (lr 1e-5/2ep), wire API to 2.7

6e1a968

ci: add fastapi + httpx to test deps (TestClient needs them)

2dcfcf8

ci: set least-privilege permissions (contents: read) on test workflow

6c03ec8

Addresses CodeQL actions hardening finding: limit GITHUB_TOKEN scope.

This was referenced Jun 26, 2026

data: FP benchmark + gap-fix training set for transactional A2P #208

Closed

SMS classification intelligence evaluation and targeted model improvement #202

Merged

ajamous self-requested a review June 28, 2026 06:08

ajamous self-assigned this Jun 28, 2026

ajamous added the enhancement New feature or request label Jun 28, 2026

ajamous changed the base branch from sms-classification-intelligence-eval to main July 2, 2026 01:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

data: FP benchmark + gap-fix training set for transactional A2P#209

data: FP benchmark + gap-fix training set for transactional A2P#209
Frhnfaya wants to merge 10 commits into
TelecomsXChangeAPi:mainfrom
Frhnfaya:feat/fp-benchmark-and-training-set

Frhnfaya commented Jun 26, 2026

Uh oh!

ajamous commented Jul 2, 2026

Uh oh!

ajamous commented Jul 2, 2026

Uh oh!

Frhnfaya commented Jul 3, 2026

Uh oh!

ajamous commented Jul 3, 2026 •

edited

Loading

Uh oh!

Frhnfaya commented Jul 4, 2026

Uh oh!

Frhnfaya commented Jul 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Frhnfaya commented Jun 26, 2026

What this adds

Baseline (model 2.7, 88-msg benchmark)

Reproduction

Uh oh!

ajamous commented Jul 2, 2026

Uh oh!

ajamous commented Jul 2, 2026

Uh oh!

Frhnfaya commented Jul 3, 2026

Uh oh!

ajamous commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Frhnfaya commented Jul 4, 2026

Uh oh!

Frhnfaya commented Jul 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ajamous commented Jul 3, 2026 •

edited

Loading