Skip to content

feat(gh-2142): add VSBC diagnostic#2170

Merged
fehiepsi merged 3 commits into
pyro-ppl:masterfrom
michaelellis003:add-vsbc-diagnostic
May 2, 2026
Merged

feat(gh-2142): add VSBC diagnostic#2170
fehiepsi merged 3 commits into
pyro-ppl:masterfrom
michaelellis003:add-vsbc-diagnostic

Conversation

@michaelellis003
Copy link
Copy Markdown
Contributor

@michaelellis003 michaelellis003 commented Apr 3, 2026

Summary

Implements the VSBC diagnostic requested in #2142.

This follows Yao et al. (2018), Section 3 / Algorithm 2: simulate datasets from the prior predictive, fit a variational approximation for each dataset, estimate marginal calibration probabilities, and test symmetry of p vs 1 - p with a two-sample KS test.

Changes

  • add vsbc_diagnostic in numpyro/infer/calibration.py
  • export vsbc_diagnostic from numpyro.infer
  • add end-to-end tests in test/infer/test_calibration.py
  • support sequential / vectorized / parallel dispatch
  • validate observed-site plumbing and simulated_data_to_args
  • reject discrete latent sample sites explicitly

References

  • Yao, Y., Vehtari, A., Simpson, D., and Gelman, A. (2018). Yes, but Did It Work?: Evaluating Variational Inference.
  • Talts, S., Betancourt, M., Simpson, D., Vehtari, A., and Gelman, A. (2018). Validating Bayesian Inference Algorithms with Simulation-Based Calibration.

Fixes #2142

Copy link
Copy Markdown
Member

@fehiepsi fehiepsi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks @michaelellis003!

Comment thread numpyro/infer/__init__.py Outdated
"log_likelihood",
"psis_diagnostic",
"reparam",
"vsbc_diagnostic",
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe not expose vsbc_diagnostic here, just add calibration here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, fixed! Thanks for the review!

@juanitorduz
Copy link
Copy Markdown
Collaborator

@michaelellis003 can you please rebase/merge with master as the fixes have been merged upstream :) ?

@juanitorduz juanitorduz mentioned this pull request May 2, 2026
@fehiepsi fehiepsi merged commit df76b46 into pyro-ppl:master May 2, 2026
5 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Variational Simulation-Based Calibration (VSBC) diagnostic

3 participants