Time-Aware Dataset Forensics & Early-Warning Risk Prediction for LLM Fine-Tuning
"Predict whether a dataset will damage an LLM — before the fine-tuning damage becomes visible."
ORIGIN-T is a final-year AIML research project that audits datasets for synthetic, recursive, paraphrased, and benchmark-leaked contamination, measures the resulting model-collapse when a small LLM is fine-tuned on that data, and learns an early-warning risk model that flags danger before fine-tuning damage is fully visible. It blends three ideas into one coherent system: ORIGIN (dataset forensics), SENTINEL-T (survival-based early warning), and IrregMamba (time-aware sequence modeling).
- Not a wrapper. No LangChain glue, no prompt-chaining product — a measurable ML research system: contamination generators, a fine-tuning grid, a feature suite, and a risk model with a defined go/no-go gate at every stage.
- Frontier-relevant. Model collapse (Shumailov, Nature 2024) and benchmark contamination are active, unsolved problems in how foundation models are trained and evaluated.
- Depth over breadth. A calibrated risk classifier is the committed core; competing-risks survival analysis, conformal guarantees, white-box representational signals, causal attribution, and a safe-mix optimizer are gated additions layered on the same experiments — not scope creep.
| # | Feature | Status |
|---|---|---|
| 1 | Controlled contamination generator (synthetic / recursive / paraphrased / benchmark-near) | Committed |
| 2 | Feature suite (lexical, entropy, perplexity, semantic, MAUVE, leakage) | Committed |
| 3 | LoRA fine-tuning grid + collapse-curve measurement | Committed |
| 4 | Calibrated risk classifier (degrade / no-degrade) | Committed |
| 5 | Split-conformal risk guarantee | Gated |
| 6 | Discrete-time survival early-warning model | Gated |
| 7 | White-box representational-collapse signals (effective rank, anisotropy) | Gated |
| 8 | Causal attribution (Shapley) + collapse scaling law | Gated |
| 9 | Safe-mix mitigation optimizer | Gated |
| 10 | ORIGIN-T-Bench — released benchmark + leaderboard | Committed (M8) |
See docs/PRD.md for full scope and docs/TRD.md for the technical design.
Raw dataset -> Normalizer -> Contamination generator -> ORIGIN-T Bench store
|
+-------------------------------+
v v
Feature extractor Fine-tuning runner
| |
Risk model <---- risk-model training --+-- Evaluation harness -> Degradation curves
|
Calibration (conformal) -> Alert / Report -> Mitigation (safe-mix)
Python 3.11 · PyTorch · HF transformers / datasets / peft /
accelerate · bitsandbytes (4-bit) · scikit-learn, XGBoost,
sentence-transformers, FAISS, mauve-text · lifelines / pycox
(survival) · MAPIE (conformal) · shap ·
TransformerLens / nnsight (interpretability) · MLflow / W&B
(offline) · Streamlit (dashboard).
OriginX-T/
README.md
docs/
PRD.md Product Requirements Document
TRD.md Technical Requirements Document
PROJECT_REPORT.md Full research report (literature, roadmap, risks)
configs/ Experiment & model configs
data/ Raw / processed datasets (large data git-ignored)
src/
ingestion/ Dataset loaders, schema inference
generation/ Synthetic + recursive contamination generation
features/ Lexical / entropy / semantic / representational features
finetune/ Small-model LoRA fine-tuning
evaluation/ Degradation metrics, benchmark evaluation
risk/ Static, survival, conformal, causal, scaling-law models
mitigation/ Data cleaning, safe-mix optimization
dashboard/ Streamlit evidence dashboard (Sprint 14)
reports/ Figures, experiment logs
tests/ Unit / smoke tests
.github/workflows/ci.yml Lint + test CI
git clone https://github.com/SheeshDarth/OriginX-T.git
cd OriginX-T
python -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r requirements.txt
pytest -q # smoke test should passGPU work (fine-tuning) runs on free Google Colab / Kaggle notebooks — see docs/TRD.md for the compute plan. A local RTX 4050 6GB GPU is sufficient for feature extraction and small-model inference.
| Role | Owns | Theory owner |
|---|---|---|
| S1 — Data & Bench Lead | Contamination pipeline, ORIGIN-T-Bench, hidden holdout | Distribution shift, information theory, MAUVE |
| S2 — Modeling Lead | Feature suite, risk models, calibration/conformal | Survival analysis, conformal prediction |
| S3 — Experiments/Infra Lead | Fine-tuning runner, eval harness, reproducibility | LoRA/PEFT, embedding geometry, mech-interp |
2-week sprints, async standups Mon/Thu, Definition of Done in CONTRIBUTING.md. Full 8-month roadmap and current sprint status: docs/PROJECT_REPORT.md.
Sprint 0 — Inception + collapse spike. See the roadmap for the current gate. Weekly updates go to the faculty guide; see the template in docs/PROJECT_REPORT.md.
MIT — permissive by default so ORIGIN-T-Bench can be released freely on HuggingFace; change before submission if your institution requires otherwise.