Skip to content

Fit describe-code metadata on training split only - #167

Draft
florian6973 wants to merge 1 commit into
mmcdermott:mainfrom
florian6973:agent/train-only-describe-metadata
Draft

Fit describe-code metadata on training split only#167
florian6973 wants to merge 1 commit into
mmcdermott:mainfrom
florian6973:agent/train-only-describe-metadata

Conversation

@florian6973

Copy link
Copy Markdown

What changed

  • fit describe-code frequencies from data/train/**/*.parquet for split-sharded MEDS datasets
  • fall back to filtering unsharded data with metadata/subject_splits.parquet
  • fail clearly when train-only fitting cannot be guaranteed
  • isolate and reduce only train-only cache files, preventing stale tuning or held-out cache entries from affecting fitted metadata
  • make Polars row orientation explicit when converting frequency dictionaries
  • add regression coverage for threshold leakage, held-out-only vocabulary entries, subject-split filtering, and missing split metadata

Why

meds-tab-describe recursively scanned all Parquet files under input_dir. On split-sharded datasets, tuning and held-out covariates therefore influenced metadata/codes.parquet and downstream feature filtering such as min_code_inclusion_count.

This is label-free held-out preprocessing leakage: evaluation covariates could affect the fitted feature vocabulary even though labels were not used.

Impact

Preprocessing metadata is now fitted only on training subjects and then reused for all splits. A code with 8 train occurrences and 5 held-out occurrences now has fitted count 8 and will not pass a minimum count of 10.

Validation

  • uv run pytest -q: 82 passed
  • targeted regression tests: 3 passed
  • Ruff check and format check passed
  • git diff --check passed

Fixes #166

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

meds-tab-describe includes tuning and held-out shards when fitting feature metadata

1 participant