Fix train/test leakage from duplicate dataset rows#195
Conversation
Remove 7,718 duplicate rows, 80 translation artifacts, and 10 junk-text rows from the v2.4 SMS dataset. Under config.py's split (seed=42, test_size=0.2), test-to-train leakage drops from 7.4% to 0%. Adds reproducible cleaning script and a test. Fixes TelecomsXChangeAPi#194.
29ff41a to
42b6347
Compare
|
@Frhnfaya thanks — I re-ran the script against the pristine v2.4 combined file and reproduced your numbers exactly (145,811 → 138,003; 7,718 dups + 80 artifacts + 10 junk; leakage 7.4% → 0% under the seed-42 split). Clean work. One thing before merge: nothing consumes the new file yet. Two smaller notes, fine as follow-ups: ~69 LLM-refusal/translation artifacts survive the marker list ("Please provide more context or clarify your request.", "Sorry, I cannot click on links…", etc.), and because the dedup key is (text, label), a few identical texts with conflicting labels survive (e.g. |
What this fixes
Fixes #194.
The combined v2.4 dataset contains duplicate rows that leak across the train/test split, inflating reported accuracy.
Findings (reproducible)
7,718 exact/normalized duplicate rows (5.4% of 145,811)
80 machine-translation artifacts mislabeled as SMS (e.g. "Sorry, I cannot provide a translation…")
10 empty/symbol-only rows
Under config.py's split (test_size=0.2, random_state=42): test→train leakage was 7.4% (2,172 rows). After cleaning: 0.0%.
Changes
clean_ots_dataset.py — reproducible cleaning utility (dedup + artifact removal + unicode normalization) that prints a per-reason audit report.
dataset/sms_spam_phishing_dataset_v2.4.1_dedup.csv — cleaned dataset (138,003 rows; label balance preserved).
test_clean_dataset.py — asserts no duplicates, valid labels, no empty text (3 passed).
CHANGELOG entry.
The original v2.4 file is left untouched; the cleaned set is added alongside it.