data: FP benchmark + gap-fix training set for transactional A2P#208
Closed
Frhnfaya wants to merge 10 commits into
Closed
data: FP benchmark + gap-fix training set for transactional A2P#208Frhnfaya wants to merge 10 commits into
Frhnfaya wants to merge 10 commits into
Conversation
…rovement Evaluate the OTS mBERT classifier against public SMS-classification benchmarks and a new multilingual adversarial suite, then close the identified gaps with a targeted synthetic dataset and an incremental fine-tune. Findings (v2.5 baseline): - 99.3% accuracy on the UCI SMS Spam Collection (classic benchmark) - 99.2% block rate on the Mishra & Soni SMS Phishing dataset, but only 2.7% phishing-label recall (smishing routed to spam) - 20.4% three-class / 69.9% block rate on the modern, multilingual IMC 2025 smishing corpus, with ~25% of phishing silently passed as ham - Non-English block rates falling to ~52-56% Improvement (v2.6 incremental fine-tune): - UCI block rate held at 99.2% (no classic-spam regression) - IMC25 block rate 69.9% -> 78.2%, phishing recall 17% -> 40% - Adversarial suite block rate 73.3% -> 98.0%, phishing recall 11.5% -> 82% - Multilingual block-rate gains concentrated where the model was weak (Italian +19.6pts, Japanese +19.7pts, Spanish +13.8pts) Adds an offline, reproducible evaluation harness, the adversarial test set, the Mishra & Soni benchmark loader, a synthetic data generator with locale-correct brands and hard-negative ham twins, an incremental fine-tune script, and a full written report (evals/REPORT.md).
…ardening Training (evals/finetune_tier1.py): - Apple Silicon MPS / CUDA / CPU device auto-detection - stratified train/validation split with per-epoch validation metrics - best-checkpoint selection by validation macro-F1 (fixes the last-epoch regressions from the proof-of-concept run) - class-weighted or focal loss to counter the phishing->spam collapse - warmup + linear LR schedule, gradient clipping Calibration (evals/calibrate_thresholds.py + run_eval --logit-bias): - training-free per-class logit-bias search to recover phishing recall; on the adversarial set this alone lifts block 73%->83% and phishing recall 12%->35% on the unchanged v2.5 weights Obfuscation hardening (production path): - strip zero-width / invisible formatting characters in the preprocessor - run homoglyph + zero-width normalisation in the dynamic-batching path (previously only the per-request path normalised, so the production default never did) Adds evals/TIER1.md runbook for reproducing on Apple Silicon.
- ModelInfo gains an 'architecture' field so 'version' is unambiguously the model semver (e.g. 2.7), not the network name; fix misleading schema examples - /health now reports both api_version (platform, e.g. 2.10.0) and a model block (name/version/architecture); 'version' kept as a backward-compatible alias - add model_manager.get_model_version() for a cheap version lookup
…obustness, normalisation tests - Untrack reproducible prediction dumps; gitignore catch-all (review TelecomsXChangeAPi#3) - Vendored BERT vocab -> Git LFS + Apache-2.0 attribution (review TelecomsXChangeAPi#1) - calibrate_thresholds: importlib load (no sys.path mutation) + with-statement (review TelecomsXChangeAPi#4,TelecomsXChangeAPi#5) - run_eval: ap.error instead of assert; DEFAULT_MODEL baseline note (review TelecomsXChangeAPi#8,TelecomsXChangeAPi#9) - finetune_incremental: --num-workers arg, default 0 (macOS safety) (review TelecomsXChangeAPi#7) - settings: OTS_MBERT_MODEL_PATH override for env-driven model rollback (review TelecomsXChangeAPi#2) - health/response_models: clarify architecture/tokenizer key; mark version deprecated (review TelecomsXChangeAPi#10,TelecomsXChangeAPi#11) - add 9 unit tests for the batching-path normalisation + fallback (review TelecomsXChangeAPi#6)
…ers module, CI, perf, tests Bugs: - settings: guard _apply_model_path_override against unknown default_mbert_version (KeyError) (TelecomsXChangeAPi#1) - run_eval: warn when scoring the v2.5 baseline default instead of production weights (TelecomsXChangeAPi#3) Quality: - batching_service: log a warning when EnhancedPreprocessor import fails (no silent disable) (TelecomsXChangeAPi#4) - extract shared dataset loaders to evals/loaders.py; run_eval + calibrate import it normally (drops importlib hack) (TelecomsXChangeAPi#5) - drop unreferenced intermediate summaries summary_v2.5_mishra/v2.6/v2.6_big.json (TelecomsXChangeAPi#6) - health: de-rot the architecture/tokenizer-key comment (no review-number ref) (TelecomsXChangeAPi#9) - enhanced_preprocessing: INVISIBLE_CHARS as \u escapes with codepoint comments (set unchanged) (minor) Performance: - batching_service: ASCII fast-path in _normalize_text skips normalisation for typical traffic (TelecomsXChangeAPi#7) API: - response_models: mark HealthResponse.version Field(deprecated=True) (TelecomsXChangeAPi#8) Tests/CI: - add GitHub Actions unit-test workflow running pytest (TelecomsXChangeAPi#12) - add settings-override tests incl. KeyError guard + per-instance isolation (TelecomsXChangeAPi#13) - add ASCII fast-path test; keep raise-swallow test on non-ASCII input (TelecomsXChangeAPi#7) Docs: - TIER1.md: loosen torch pin to stable; flag in-domain validation caveat (minor) - dataset/README_SYNTHETIC.md: mark synthetic data, warn against pipeline misuse (TelecomsXChangeAPi#11)
Addresses CodeQL actions hardening finding: limit GITHUB_TOKEN scope.
- load_imc25: skip and count rows with no scam_type instead of silently labelling them phishing; document IMC25 as an all-attack corpus (no ham class), confirming the spam/phishing mapping is correct for it - use md5(usedforsecurity=False) for the deterministic placeholder seed so the loader works under FIPS-enforced environments - REPORT.md: note IMC25 measures attack recall only (no false-positive signal), flag the missing multilingual legitimate-message control as the top next addition, and mark the batching-path obfuscation normalization as done rather than pending
- ham_benchmark_v1.csv: 88-msg held-out benchmark (en/es/it/pt, 8 categories) - ham_training_v1.csv: 486-row training set (282 ham + 204 phishing hard-negatives) - build_benchmark.py / build_training.py: reproducible generators - README_fp_fix.md: problem, baseline numbers, usage - summary_baseline.json: model 2.7 baseline (67% FP overall, government 94%) Zero overlap between benchmark and training set (leakage-free). Closes the gap identified in TelecomsXChangeAPi#202 comments: transactional A2P mislabelled spam.
Author
|
Superseded by #209 (wrong base branch). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows up on the false-positive discussion in #202.
What this adds
evals/datasets/ham_benchmark_v1.csv— 88-msg held-out benchmark (en/es/it/pt,8 categories). Block rate on this set = false-positive rate. Locked — never train on it.
evals/datasets/ham_training_v1.csv— 486-row training set: 282 realistic brandedA2P ham + 204 hard-negative scam twins (labelled phishing). Gap-weighted toward
government and delivery where the FP rate is worst. Zero overlap with the benchmark.
build_benchmark.py/build_training.py— reproducible generators.README_fp_fix.md— problem, root cause, baseline numbers, usage.evals/results/summary_baseline.json— model 2.7 baseline evidence.Baseline (model 2.7, 88-msg benchmark)
government94%,delivery92%,appointment/telecom75%,bank_alert67%personal/personal_money0% — conversational ham is cleanReproduction steps
Expected: overall FP 67%, government 94%, delivery 92%.
Honest caveat: both sets are constructed-realistic, not scraped. The before/after
proves the model can learn the A2P-is-ham boundary when given the data; real operator
A2P would be the gold-standard confirmation on top.