Skip to content

fix(formula): bin pandas Categorical columns in i() - #2037

Draft
mayuriphad wants to merge 2 commits into
py-econometrics:masterfrom
mayuriphad:fix/i-bin-categorical
Draft

mayuriphad wants to merge 2 commits into
py-econometrics:masterfrom
mayuriphad:fix/i-bin-categorical

Conversation

@mayuriphad

@mayuriphad mayuriphad commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #1910. i(x, bin=...) now bins a pandas Categorical column the way it bins a string column. _apply_binning used Series.replace, which cannot introduce a new category, so it raised under pandas 3 and warned under pandas 2. For a Categorical it now rebuilds the categories, and each bin takes the place of the first value listed for it, as in fixest's bin_factor. String and numeric columns still go through replace unchanged.

df["x"] = pd.Categorical(values, categories=["d", "c", "b", "a"])
pf.feols("Y ~ i(x, bin={'ab': ['a', 'b']})", df)
# before (pandas 3): TypeError: Cannot setitem on a Categorical with a new category (ab), set the categories first
# before (pandas 2): FutureWarning: The behavior of Series.replace ... with CategoricalDtype is deprecated
# after: coefficients Intercept, x::c, x::ab (levels d, c, ab; reference d)

Verification

Passed: tests/test_others.py -k bin (4 passed) under both pandas 2.3.3 and pandas 3.1.1. That includes the new test_i_bin_categorical for unordered and ordered categoricals, which fails on master under pandas 3. ruff check and ruff format --check are clean on the changed files.

Not run locally: tests/test_i.py::test_binning_categorical, the new comparison against fixest (non-alphabetical categories, with and without a continuous interaction). It needs R and rpy2, which I don't have here, so CI is the evidence for it. I tested on Windows with the compiled _core_impl from the 0.70.0 wheel over this checkout, without pixi.

Signed-off-by: mayuriphad <mayuriphad656@gmail.com>
@codecov

codecov Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Flag Coverage Δ
core-tests 88.39% <100.00%> (+0.01%) ⬆️
test-r-core-other 63.51% <100.00%> (+0.03%) ⬆️
test-r-extended 27.66% <0.00%> (-0.03%) ⬇️
test-r-fixest 48.97% <0.00%> (?)
tests-extended ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...stimation/formula/transforms/factor_interaction.py 98.51% <100.00%> (+0.08%) ⬆️

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

fixest's bin_factor keeps a bin at the position of the first value listed
in bin, not at the first binned level. With categories cherry, banana,
apple and bin fruit = apple, cherry the levels are now banana, fruit,
which matches R in test_binning_categorical.

Signed-off-by: mayuriphad <mayuriphad656@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support bin= in i() for pandas Categorical columns

1 participant