Skip to content

Fix LabelEncoder/OneHotEncoder on category columns with missing values - #1062

Open
techi101 wants to merge 1 commit into
uber:masterfrom
techi101:fix/encoder-category-nan
Open

techi101 wants to merge 1 commit into
uber:masterfrom
techi101:fix/encoder-category-nan

Conversation

@techi101

@techi101 techi101 commented Nov 7, 2026

Copy link
Copy Markdown

Proposed changes

LabelEncoder and OneHotEncoder in causalml/features.py fill missing values with x.fillna(NAN_INT). On a pandas category column that raises:

TypeError: Cannot setitem on a Categorical with a new category (-98765), set the categories first

So both encoders crash on a category column with any missing value, while the same data as a plain string column works. This happens in fit, transform and fit_transform.

The fix adds a small helper that casts a category column to object before the fill, and uses it at the three places that call fillna(NAN_INT). Output for a category column now matches the output for the same values as strings.

Types of changes

  • Bugfix (non-breaking change which fixes an issue)

Checklist

  • I have read the CONTRIBUTING doc
  • I have signed the CLA
  • Lint and unit tests pass locally with my changes
  • I have added tests that prove my fix is effective or that my feature works

Further comments

New parametrized test in tests/test_features.py for both encoders: a category column with NaN encodes the same as the string column, through fit_transform and through fit then transform. It fails on master and passes with the change. black --check passes.

I used Claude to help find and fix this, and reviewed and tested the change myself.

fillna(NAN_INT) raises TypeError on a pandas category column, since
NAN_INT is not one of its categories. Cast category columns to object
before filling. Add tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant