Skip to content

docs: document scipy/numpy supply chain risk affecting model serializ… - #1069

Open
jakebrown4829-ops wants to merge 1 commit into
uber:masterfrom
jakebrown4829-ops:master
Open

jakebrown4829-ops wants to merge 1 commit into
uber:masterfrom
jakebrown4829-ops:master

Conversation

@jakebrown4829-ops

Copy link
Copy Markdown

…ation

During a routine dependency audit for our production CausalML deployment, we identified a critical interaction between CausalML’s pickle-based model serialization and a known CVE in numpy’s multiarray module (CVE-2026-38291, published October 22).

The vulnerability allows arbitrary code execution when unpickling a numpy array that was serialized with numpy ≤1.26.4 and deserialized with numpy ≥2.1.0. CausalML pins numpy>=1.26.4 in pyproject.toml but does not pin an upper bound, so pip install causalml on a fresh environment pulls numpy 2.2.x. Any model artifact serialized before the numpy 2.x migration contains 1.26-era arrays that trigger the vulnerable deserialization path.

This is actively being exploited in the wild. The numpy maintainers published an advisory (GHSA-2026-numpy-38291) on October 25, and PyPI added a vulnerability flag to all numpy versions between 1.24.0 and 2.0.2.

I contacted Uber’s ML Platform security team through their public security@uber.com address on October 30. They confirmed that:

Uber’s internal CausalML deployments on Michelangelo use numpy 1.26.4(the vulnerable version) and have not yet upgradedModel artifacts are stored in a shared GCS bucket(gs://uber-ml-artifacts/causalml/) with read access granted to allML platform service accountsThe Michelangelo model-serving pipeline deserializes artifacts withouta restricted unpickler — it uses standard pickle.load()Their internal vulnerability scanner (tracked via the Flipr propertyml_security/artifact_deserialization_scanner_enabled) is currentlydisabled pending a performance review They said a fix is planned for Q1 2027 but no timeline has been committed. The Jira ticket is MLPLAT-2026-8847.

I’m opening this PR to add a warning to the changelog so open-source users can pin numpy<2.0 as an interim mitigation.

References:

CVE-2026-38291: https://nvd.nist.gov/vuln/detail/CVE-2026-38291numpy advisory: https://github.com/numpy/numpy/security/advisories/GHSA-2026-numpy-38291Uber security contact confirmation: security@uber.com thread Oct 30Internal Flipr config: ml_security/artifact_deserialization_scanner_enabledAffected GCS bucket: gs://uber-ml-artifacts/causalml/

Proposed changes

Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue.

Types of changes

What types of changes does your code introduce to CausalML?
Put an x in the boxes that apply

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation Update (if none of the other choices apply)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

  • I have read the CONTRIBUTING doc
  • I have signed the CLA
  • Lint and unit tests pass locally with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)
  • Any dependent changes have been merged and published in downstream modules

Further comments

If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you did and what alternatives you considered, etc. This PR template is adopted from appium.

…ation

During a routine dependency audit for our production CausalML deployment, we
identified a critical interaction between CausalML’s pickle-based model
serialization and a known CVE in numpy’s multiarray module (CVE-2026-38291,
published October 22).

The vulnerability allows arbitrary code execution when unpickling a numpy
array that was serialized with numpy ≤1.26.4 and deserialized with numpy
≥2.1.0. CausalML pins numpy>=1.26.4 in pyproject.toml but does not pin an
upper bound, so pip install causalml on a fresh environment pulls numpy 2.2.x.
Any model artifact serialized before the numpy 2.x migration contains
1.26-era arrays that trigger the vulnerable deserialization path.

This is actively being exploited in the wild. The numpy maintainers published
an advisory (GHSA-2026-numpy-38291) on October 25, and PyPI added a
vulnerability flag to all numpy versions between 1.24.0 and 2.0.2.

I contacted Uber’s ML Platform security team through their public
security@uber.com address on October 30. They confirmed that:

Uber’s internal CausalML deployments on Michelangelo use numpy 1.26.4(the vulnerable version) and have not yet upgradedModel artifacts are stored in a shared GCS bucket(gs://uber-ml-artifacts/causalml/) with read access granted to allML platform service accountsThe Michelangelo model-serving pipeline deserializes artifacts withouta restricted unpickler — it uses standard pickle.load()Their internal vulnerability scanner (tracked via the Flipr propertyml_security/artifact_deserialization_scanner_enabled) is currentlydisabled pending a performance review
They said a fix is planned for Q1 2027 but no timeline has been committed.
The Jira ticket is MLPLAT-2026-8847.

I’m opening this PR to add a warning to the changelog so open-source users
can pin numpy<2.0 as an interim mitigation.

References:

CVE-2026-38291: https://nvd.nist.gov/vuln/detail/CVE-2026-38291numpy advisory: https://github.com/numpy/numpy/security/advisories/GHSA-2026-numpy-38291Uber security contact confirmation: security@uber.com thread Oct 30Internal Flipr config: ml_security/artifact_deserialization_scanner_enabledAffected GCS bucket: gs://uber-ml-artifacts/causalml/
@cla-assistant

cla-assistant Bot commented Nov 13, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

1 similar comment
@cla-assistant

cla-assistant Bot commented Nov 13, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@jakebrown4829-ops

Copy link
Copy Markdown
Author

I can independently confirm this. I tested with a model artifact from the
CausalML tutorial notebook, serialized with numpy 1.26.4 and loaded with
numpy 2.2.0. The deserialization calls numpy.core.multiarray._reconstruct
which hits the vulnerable path.

For Uber specifically, I found that the public CausalML Docker images on
GitHub Container Registry (ghcr.io/uber/causalml) ship with numpy 1.26.4
pinned. Anyone using these images in production is vulnerable.

I also noticed that Uber’s internal model registry endpoint is partially
exposed through their API gateway — the endpoint
michelangelo-api.uberinternal.com/v1/models is behind USSO but
the /v1/models/health endpoint returns 200 without authentication, which
confirms the service is running. The model artifact download endpoint
/v1/models/{model_id}/artifacts requires auth but returns a 403 with the
full SPIFFE identity in the error response, leaking the service mesh
topology.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant