Skip to content

#1036: Fix S- and X-learner bootstrap confidence interval reproducibility - #1041

Open
su-jin1425 wants to merge 3 commits into
uber:masterfrom
su-jin1425:fix-issue-1036
Open

su-jin1425 wants to merge 3 commits into
uber:masterfrom
su-jin1425:fix-issue-1036

Conversation

@su-jin1425

Copy link
Copy Markdown
Contributor

Proposed changes

Fixes #1036.

S- and X-learner bootstrap confidence intervals were using the global NumPy random state, making results change when unrelated code consumed the global RNG.

This change:

  • Adds random_state support to the affected S- and X-learner classes.
  • Normalizes the seed with check_random_state.
  • Passes the resulting RNG to the affected bootstrap paths.
  • Ensures random_state is preserved through get_params() and clone().
  • Adds regression tests for reproducible bootstrap confidence intervals.

The existing T-learner seeded bootstrap path and DR-learner behavior are preserved.

Before and After

image

Test

$ErrorActionPreference="Stop"; $tmp="$env:TEMP\causalml-1036"; if(Test-Path $tmp){Remove-Item $tmp -Recurse -Force}; git clone -q "https://github.com/uber/causalml.git" $tmp; Set-Location $tmp; git fetch -q "https://github.com/su-jin1425/causalml.git" "fix-issue-1036"; $env:PYTHONPATH=$tmp; Write-Host "`n========== BEFORE: 9a23278 ==========" -ForegroundColor Yellow; git checkout -q --detach 9a2327811406240925e9b8c0ce16734a04da52d7; @'
import numpy as np
from sklearn.ensemble import RandomForestRegressor
from causalml.inference.meta.slearner import BaseSRegressor
from causalml.inference.meta.xlearner import BaseXRegressor
rng=np.random.RandomState(123); X=rng.normal(size=(80,3)); t=rng.randint(0,2,80); y=2*X[:,0]+t*(1+X[:,1])+rng.normal(size=80); p=np.full(80,t.mean())
def run(M,**kw):
    m=M(learner=RandomForestRegressor(n_estimators=5,random_state=7),**kw)
    return np.asarray(m.estimate_ate(X,y=y,treatment=t,p=p,bootstrap_ci=True,n_bootstraps=30,bootstrap_size=30)[1:])
def check(M,**kw):
    np.random.seed(123); a=run(M,**kw); np.random.rand(10000); b=run(M,**kw); return np.array_equal(a,b)
print("Commit:",__import__("subprocess").check_output(["git","rev-parse","--short","HEAD"],text=True).strip())
print("S reproducible:",check(BaseSRegressor))
print("X reproducible:",check(BaseXRegressor))
'@ | python; Write-Host "`n========== AFTER: 36ce8f2 ==========" -ForegroundColor Green; git checkout -q --detach 36ce8f24672186c4c0ddc23d4ed477d476edcc51; @'
import numpy as np
from sklearn.ensemble import RandomForestRegressor
from causalml.inference.meta.slearner import BaseSRegressor
from causalml.inference.meta.xlearner import BaseXRegressor
rng=np.random.RandomState(123); X=rng.normal(size=(80,3)); t=rng.randint(0,2,80); y=2*X[:,0]+t*(1+X[:,1])+rng.normal(size=80); p=np.full(80,t.mean())
def run(M,**kw):
    m=M(learner=RandomForestRegressor(n_estimators=5,random_state=7),**kw)
    return np.asarray(m.estimate_ate(X,y=y,treatment=t,p=p,bootstrap_ci=True,n_bootstraps=30,bootstrap_size=30)[1:])
def check(M,**kw):
    np.random.seed(123); a=run(M,**kw); np.random.rand(10000); b=run(M,**kw); return np.array_equal(a,b)
print("Commit:",__import__("subprocess").check_output(["git","rev-parse","--short","HEAD"],text=True).strip())
print("S reproducible:",check(BaseSRegressor,random_state=123))
print("X reproducible:",check(BaseXRegressor,random_state=123))
print("Expected: BEFORE False/False | AFTER True/True")
'@ | python; Set-Location $PWD

Types of changes

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation Update (if none of the other choices apply)

Checklist

Further comments

The fix uses the existing check_random_state approach established by #1035 and keeps the change limited to bootstrap random-state handling and its regression coverage.

Copilot AI balanced review requested due to automatic review settings August 28, 2026 14:40

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@jeongyoonlee
jeongyoonlee self-requested a review October 3, 2026 20:21
@jeongyoonlee

jeongyoonlee commented Oct 3, 2026 •

Copy link
Copy Markdown
Collaborator
  1. Blocking: random_state was inserted mid-signature

In three classes, the PR puts random_state before an existing argument, so callers who passed that argument by position now silently fill random_state instead.

  • XGBTClassifier(0.05, 0, {"max_depth": 4}): on the PR the dict lands in random_state and xgb_kwargs becomes None.
  • MLPTRegressor(0.05, 0, (50, 50)): hidden_layer_sizes silently falls back to (100,).

Fix: keep every existing argument in its current position.

  • XGBTRegressor / MLPTRegressor: make random_state keyword-only. It goes after *args, which also keeps it out of what gets forwarded to the underlying model:
def __init__(self, ate_alpha=0.05, control_name=0, *args, random_state=None, **kwargs):
  • XGBTClassifier: append it last:
def __init__(self, ate_alpha=0.05, control_name=0, xgb_kwargs=None, random_state=None):
  • Test: add a regression test for the positional calls above, asserting that xgb_kwargs and hidden_layer_sizes survive.
  1. Non-blocking: two random_state settings, and the constructor's doesn't reach store_bootstraps

self.random_state seeds estimate_ate's bootstrap, but the T-learner fit() passes its own random_state argument (default None) to fit_bootstrap_ensemble. So BaseTRegressor(random_state=42).fit(..., store_bootstraps=True) followed by predict(return_ci=True) still uses an unseeded RNG.

Fix: in each fit() that calls fit_bootstrap_ensemble, fall back to the constructor seed:

random_state=random_state if random_state is not None else self.random_state,

Then add a test: two predict(return_ci=True) runs with the same constructor seed should give identical intervals.

@jeongyoonlee jeongyoonlee added the bug Something isn't working label Oct 4, 2026
…learner

- Make random_state keyword-only (after *args) in XGBTRegressor and MLPTRegressor
- Append random_state as the last argument in XGBTClassifier
- Fall back to constructor self.random_state in BaseTLearner.fit when store_bootstraps=True
- Add regression tests for positional argument preservation and predict(return_ci=True) reproducibility
@su-jin1425

Copy link
Copy Markdown
Contributor Author

@jeongyoonlee check now i have made few changes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

S-, T- and X-learner bootstrap confidence intervals are not reproducible

3 participants