cleanllm

Streaming JSONL cleaner for LLM fine-tuning datasets. Minimal dependencies, memory-safe, and fast — processes files line-by-line without loading them into memory.

What it does

cleanllm gives you a pipeline for cleaning, validating, and profiling JSONL datasets before fine-tuning:

raw.jsonl → scan → fix → dedup → validate → stats → audit bundle → shards

Every step is streaming (no full-file load), resumable, and produces machine-readable JSON reports for CI gating.

Install

pip install cleanllm

Or from source:

git clone https://github.com/verma8076/cleanllm
cd cleanllm
pip install -e .

For GPU-accelerated IFD scoring and semantic deduplication:

pip install cleanllm[gpu]       # real IFD scoring (GPT-2 forward passes)
pip install cleanllm[semantic]  # semantic dedup (sentence-transformers)
pip install cleanllm[gpu,semantic]  # both

GPU Benchmark

Measured on Google Colab Tesla T4 (15.6 GB VRAM), 500 records, gpt2 (117M params):

Feature	Device	rec/sec
IFD scoring (`ifd-score`)	CPU	2.9
IFD scoring (`ifd-score`)	T4 GPU	43.1 — 14.9× faster
IFD scoring via CLI file	CPU	2.6
IFD scoring via CLI file	T4 GPU	32.1 — 12.3× faster
Semantic dedup	CPU	25.3
Semantic dedup	T4 GPU	68.6 — 2.7× faster

Semantic model: all-MiniLM-L6-v2. Run benchmark_colab.ipynb to reproduce.

Quickstart

# Scan for issues
cleanllm scan data.jsonl

# Fix: remove URLs, normalize whitespace, redact forbidden patterns
cleanllm fix data.jsonl -o data.cleaned.jsonl

# Deduplicate by prompt content
cleanllm dedup data.cleaned.jsonl -o data.dedup.jsonl --by prompt

# Profile the cleaned dataset
cleanllm stats data.dedup.jsonl --report-json stats.json

# Gate in CI: fail if invalid rows increased
cleanllm gate --compare compare.json --rules gate_rules.json

CLI reference

`scan`

Streaming scan for issues — invalid JSON, missing keys, URLs, forbidden patterns, language distribution, duplicate estimate.

cleanllm scan data.jsonl
cleanllm scan data.jsonl --report-json scan_report.json --dup-estimate
cleanllm scan data.jsonl --preset cp_portable

`fix`

Remove URLs, normalize whitespace, redact or drop rows with forbidden patterns.

cleanllm fix data.jsonl -o cleaned.jsonl
cleanllm fix data.jsonl -o cleaned.jsonl --drop-on forbidden_pattern --drop-on invalid_json
cleanllm fix data.jsonl -o cleaned.jsonl --preset cpp17_clean --report-json fix_report.json

Drop rules: invalid_json, missing_required_keys, forbidden_pattern, empty_assistant, placeholder, repetitive_response, bad_conversation.

Note on empty_assistant: By default this drops assistant responses shorter than 20 characters — calibrated for code datasets where very short responses are almost always errors. For text/chat datasets, set --min-assistant-chars 1 to only drop truly blank responses.

`validate`

Schema validation, line by line. Exit code 0 only if all rows pass.

cleanllm validate data.jsonl --schema basic_sft
cleanllm validate data.jsonl --schema cp_sft_v1

Schema	Required fields
`basic_sft`	`id`, `messages` (list of `role`/`content` dicts)
`cp_sft_v1`	`id`, `source`, `problem_id`, `messages`, `tests` (non-empty, with `input`/`output`)

`dedup`

First-occurrence deduplication — by full record, prompt (system+user), or code (assistant).

cleanllm dedup data.jsonl -o deduped.jsonl --by record
cleanllm dedup data.jsonl -o deduped.jsonl --by prompt --normalized
cleanllm dedup data.jsonl -o deduped.jsonl --by code --report-json dedup_report.json

`stats`

Single-pass profiler: distributions, structural stats, schema counts, response lengths, language distribution.

cleanllm stats data.jsonl
cleanllm stats data.jsonl --schema cp_sft_v1 --keys source,difficulty_bucket --top-k 20
cleanllm stats data.jsonl --report-json stats.json

`compare`

Diff two stats reports to catch regressions between dataset versions.

cleanllm compare old_stats.json new_stats.json
cleanllm compare old_stats.json new_stats.json --report-json compare.json
cleanllm compare old.jsonl new.jsonl --from-jsonl --schema cp_sft_v1

`gate`

CI-friendly quality gating. Nonzero exit on failures.

cleanllm gate --stats stats.json --rules gate_rules.json
cleanllm gate --compare compare.json --rules gate_rules.json --strict
cleanllm gate --compare compare.json --inline-rule "counts_diff.invalid_json_rows.delta<=0"

Gate rules JSON:

{
  "version": 1,
  "mode": "compare",
  "rules": [
    {"name": "no_new_invalid", "metric": "counts_diff.invalid_json_rows.delta", "op": "<=", "value": 0},
    {"name": "enough_valid",   "metric": "counts_diff.valid_json_rows.new",    "op": ">=", "value": 1000}
  ]
}

Supported ops: ==, !=, <, <=, >, >=. Severities: error (default), warn.

`run`

Execute a JSON-defined multi-step pipeline with variable substitution.

cleanllm run --config pipeline.json
cleanllm run --config pipeline.json --set input_path=data.jsonl --set outdir=out/v2
cleanllm run --config pipeline.json --dry-run

Supported step types: fix, validate, dedup, stats, audit, sample, shard, manifest, scan, compare.

`sample`

Reservoir sampling — random or stratified, deterministic with --seed.

cleanllm sample data.jsonl -o sample.jsonl -n 500 --seed 42
cleanllm sample data.jsonl -o sample.jsonl -n 500 --stratify source,difficulty_bucket

`audit`

Build a reproducible audit bundle in one command: sampled JSONL + CSV review index (with original line numbers) + summary + manifest.

cleanllm audit data.jsonl --outdir audit_bundle -n 200 --seed 42
cleanllm audit data.jsonl --outdir audit_bundle -n 200 --stratify source --schema cp_sft_v1

Bundle contents: audit_sample.jsonl, audit_index.csv, audit_summary.json, AUDIT_README.md, manifest.json.

`shard` / `manifest`

cleanllm shard data.jsonl --outdir shards --size 5000 --gzip
cleanllm manifest shards -o manifest.json

`convert`

Convert a JSONL file between sharegpt, alpaca, and chatml formats.

cleanllm convert data.jsonl -o converted.jsonl --from sharegpt --to chatml
cleanllm convert data.jsonl -o converted.jsonl --from alpaca --to sharegpt

Supported formats: sharegpt (conversations list), alpaca (instruction/output), chatml (messages list).

`merge`

Merge multiple JSONL files into one, with optional deduplication.

cleanllm merge a.jsonl b.jsonl c.jsonl -o merged.jsonl
cleanllm merge a.jsonl b.jsonl -o merged.jsonl --dedup

`split`

Split a JSONL file into train and val sets.

cleanllm split data.jsonl --outdir splits/
cleanllm split data.jsonl --outdir splits/ --ratio 0.95 --seed 42 --no-shuffle

Outputs <basename>_train.jsonl and <basename>_val.jsonl in the output directory. Default ratio is 0.9 (90% train).

`recipes`

Bootstrap pipelines and gate rules from built-in templates.

cleanllm recipes list
cleanllm recipes show cp_pipeline_cp_portable
cleanllm recipes write cp_bundle --outdir bootstrap/

Built-in recipes: cp_pipeline_basic, cp_pipeline_cp_portable, cp_pipeline_fast_audit, gate_stats_basic, gate_compare_basic, gate_compare_strict, cp_bundle.

Python API

from cleanllm import (
    scan_jsonl, fix_jsonl, FixRules,
    dedup_jsonl, validate_jsonl, stats_jsonl,
    sample_jsonl, audit_bundle,
    shard_jsonl, make_manifest,
    download_from_hub, detect_hf_schema,
)
from cleanllm.convert import convert_jsonl
from cleanllm.merge import merge_jsonl
from cleanllm.split import split_jsonl

# Scan
report = scan_jsonl("data.jsonl")

# Fix (code dataset)
rules = FixRules(
    drop_on={"forbidden_pattern", "empty_assistant"},
    max_tokens=4096,
    keep_language="python",
)
summary = fix_jsonl("data.jsonl", "cleaned.jsonl", rules)

# Fix (text/chat dataset — only drop truly blank responses)
rules = FixRules(drop_on={"empty_assistant"}, min_assistant_chars=1, forbidden_patterns=[])

# Dedup
result = dedup_jsonl("cleaned.jsonl", "deduped.jsonl", by="prompt", normalized=True)

# Stats
stats = stats_jsonl("deduped.jsonl", schema="cp_sft_v1", keys=["source", "difficulty_bucket"])

# Sample + audit
sample_jsonl("deduped.jsonl", "sample.jsonl", num_rows=200, seed=42)
audit_bundle("deduped.jsonl", "audit_bundle", num_rows=200, seed=42, stratify=["source"])

# Shard + manifest
shard_jsonl("deduped.jsonl", "shards", shard_size=5000, gzip_output=True)
make_manifest("shards", "manifest.json")

# Convert between formats
convert_jsonl("data.jsonl", "out.jsonl", from_fmt="sharegpt", to_fmt="chatml")

# Merge + split
merge_jsonl(["a.jsonl", "b.jsonl"], "merged.jsonl", dedup=True)
split_jsonl("merged.jsonl", "splits/", ratio=0.9, seed=42)

# Download from HuggingFace Hub (requires pip install cleanllm[hf])
result = download_from_hub("HuggingFaceH4/ultrachat_200k", "data.jsonl", split="train_sft")

Presets

Preset	Description
`general`	URL removal + whitespace normalization, no domain-specific forbidden patterns
`security_scan`	Redacts secrets: AWS keys, GitHub tokens, API keys, private keys
`pii_scan`	Redacts PII: emails, US phone numbers, SSNs, credit cards, IPv4 addresses
`cpp17_clean`	URL removal + whitespace normalization + redact C++ portability issues
`cp_portable`	Strict CP portability — drops rows with forbidden patterns
`deterministic_only`	Drops rows with non-deterministic APIs (`rand()`, `random_device`, etc.)

Defaults

Required keys: id, messages
Forbidden patterns (default): none — use --preset cpp17_clean or --preset cp_portable for CP datasets
empty_assistant threshold: 20 characters (responses shorter than this are flagged as empty)

CP datasets: To apply competitive-programming forbidden patterns (freopen, ifstream, bits/extc++.h, etc.) use a preset: cleanllm fix data.jsonl -o out.jsonl --preset cp_portable. In Python, pass forbidden_patterns=list(DEFAULT_FORBIDDEN_PATTERNS) explicitly.

Data format

cleanllm expects JSONL where each line is a JSON object. The default schema (cp_sft_v1) requires:

{
  "id": "unique-id",
  "messages": [
    {"role": "system",    "content": "..."},
    {"role": "user",      "content": "..."},
    {"role": "assistant", "content": "..."}
  ]
}

Optional fields: source, difficulty_bucket, problem_id, tests.

Development

pip install -e .[dev]
pytest
python -m build
twine check dist/*

See RELEASE_CHECKLIST.md for the full release workflow.

License

MIT

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
.vscode		.vscode
release/v0.1.0		release/v0.1.0
scripts		scripts
src		src
test_datasets		test_datasets
test_results		test_results
tests		tests
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
README.md		README.md
RELEASE_CHECKLIST.md		RELEASE_CHECKLIST.md
RELEASE_NOTES_TEMPLATE.md		RELEASE_NOTES_TEMPLATE.md
RELEASE_WORKFLOW.md		RELEASE_WORKFLOW.md
TEST_SUMMARY.md		TEST_SUMMARY.md
benchmark_colab.ipynb		benchmark_colab.ipynb
ec2_pipeline.py		ec2_pipeline.py
fix_integration_tests.py		fix_integration_tests.py
fix_tests.py		fix_tests.py
gen_datasets.py		gen_datasets.py
print_results.py		print_results.py
pyproject.toml		pyproject.toml
report_test.json		report_test.json
run_tests.py		run_tests.py
test_output_cleaned.jsonl		test_output_cleaned.jsonl

Navigation Menu

Repository files navigation

cleanllm

What it does

Install

GPU Benchmark

Quickstart

CLI reference

`scan`

`fix`

`validate`

`dedup`

`stats`

`compare`

`gate`

`run`

`sample`

`audit`

`shard` / `manifest`

`convert`

`merge`

`split`

`recipes`

Python API

Presets

Defaults

Data format

Development

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

cleanllm

What it does

Install

GPU Benchmark

Quickstart

CLI reference

scan

fix

validate

dedup

stats

compare

gate

run

sample

audit

shard / manifest

convert

merge

split

recipes

Python API

Presets

Defaults

Data format

Development

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`scan`

`fix`

`validate`

`dedup`

`stats`

`compare`

`gate`

`run`

`sample`

`audit`

`shard` / `manifest`

`convert`

`merge`

`split`

`recipes`