Skip to content

Fix train/test leakage from duplicate dataset rows#195

Open
Frhnfaya wants to merge 1 commit into
TelecomsXChangeAPi:mainfrom
Frhnfaya:fix/dataset-deduplication
Open

Fix train/test leakage from duplicate dataset rows#195
Frhnfaya wants to merge 1 commit into
TelecomsXChangeAPi:mainfrom
Frhnfaya:fix/dataset-deduplication

Conversation

@Frhnfaya

Copy link
Copy Markdown

What this fixes
Fixes #194.
The combined v2.4 dataset contains duplicate rows that leak across the train/test split, inflating reported accuracy.
Findings (reproducible)

7,718 exact/normalized duplicate rows (5.4% of 145,811)
80 machine-translation artifacts mislabeled as SMS (e.g. "Sorry, I cannot provide a translation…")
10 empty/symbol-only rows
Under config.py's split (test_size=0.2, random_state=42): test→train leakage was 7.4% (2,172 rows). After cleaning: 0.0%.

Changes

clean_ots_dataset.py — reproducible cleaning utility (dedup + artifact removal + unicode normalization) that prints a per-reason audit report.
dataset/sms_spam_phishing_dataset_v2.4.1_dedup.csv — cleaned dataset (138,003 rows; label balance preserved).
test_clean_dataset.py — asserts no duplicates, valid labels, no empty text (3 passed).
CHANGELOG entry.

The original v2.4 file is left untouched; the cleaned set is added alongside it.

Remove 7,718 duplicate rows, 80 translation artifacts, and 10 junk-text rows from the v2.4 SMS dataset. Under config.py's split (seed=42, test_size=0.2), test-to-train leakage drops from 7.4% to 0%. Adds reproducible cleaning script and a test. Fixes TelecomsXChangeAPi#194.
@ajamous

ajamous commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

@Frhnfaya thanks — I re-ran the script against the pristine v2.4 combined file and reproduced your numbers exactly (145,811 → 138,003; 7,718 dups + 80 artifacts + 10 junk; leakage 7.4% → 0% under the seed-42 split). Clean work.

One thing before merge: nothing consumes the new file yet. config.py still points current_dataset at v2.4_combined.csv, and even the glob fallback would keep picking the old file (. sorts before _, so v2.4.1_dedup lands earlier in sorted()). Since the PR says "Fixes #194", merging as-is would close the issue while every retrain keeps using the leaky dataset. Either flip current_dataset to the dedup file in this PR, or drop the "Fixes" keyword and we track the switch + retrain as a follow-up — your call, both work for me.

Two smaller notes, fine as follow-ups: ~69 LLM-refusal/translation artifacts survive the marker list ("Please provide more context or clarify your request.", "Sorry, I cannot click on links…", etc.), and because the dedup key is (text, label), a few identical texts with conflicting labels survive (e.g. faceb00k.com as both spam and phishing).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dataset v2.4 has duplicate rows causing train/test leakage

2 participants