Skip to content

Respect num_samples and normalize weights in weighted extract - #237

Open
parth-patel010 wants to merge 1 commit into
arviz-devs:mainfrom
parth-patel010:fix/extract-weighted-num-samples
Open

parth-patel010 wants to merge 1 commit into
arviz-devs:mainfrom
parth-patel010:fix/extract-weighted-num-samples

Conversation

@parth-patel010

Copy link
Copy Markdown

Fixes #236

extract(..., weights=...) with the default stratified resampling ignored num_samples and used the weights as given, so weights that didn't sum to 1 collapsed the resample onto the first few draws (or hit an IndexError).

Changes:

  • extract casts the weights to float, raises a ValueError for negative, non-finite or all-zero weights, and normalizes them once. That covers both the stratified and multinomial paths. Multinomial used to raise "Probabilities do not sum to 1" for unnormalized weights.
  • _stratified_resample takes num_samples and draws that many stratified uniforms. The Python while loop is replaced with np.searchsorted(..., side="right"), which picks the same "first index whose cumulative weight is above u" and is clipped to guard against rounding at the end of the cumulative sum.
  • Docstring for weights says they are normalized internally. reorg.pyi was regenerated with docstub run src/arviz_base.

With normalized weights and num_samples == len(weights) the selected indices are identical to before. I checked 300 random weight vectors (sizes 5 to 3000, some leading zero weights) against the old loop with the same seed: 0 mismatches. So power_scale_dataset in arviz-stats gives the same results. tests/test_psense.py and tests/test_metrics.py in arviz-stats pass against this branch (80 passed).

Repro from the issue

Before (main, b87afe3):

requested 500, got 4000
unique draws kept: 22 of 4000
resampled mean: 0.481 (expected ~1)
multinomial: requested 500, got 500

After (this branch):

requested 500, got 500
unique draws kept: 2368 of 4000
resampled mean: 0.959 (expected ~1)
multinomial: requested 500, got 500

Tests

New tests in tests/test_reorg.py::TestExtract: num_samples is respected (both methods), unnormalized weights give the same draws as the normalized ones (both methods), stratified counts match the weights exactly (weights 1:2:1 on three draws with 40 samples gives 10/20/10), and invalid weights raise.

On main (src reverted, new tests kept):

FAILED tests/test_reorg.py::TestExtract::test_weights_num_samples[stratified]
FAILED tests/test_reorg.py::TestExtract::test_weights_unnormalized[stratified]
FAILED tests/test_reorg.py::TestExtract::test_weights_unnormalized[multinomial]
FAILED tests/test_reorg.py::TestExtract::test_weights_stratified_follow_weights
FAILED tests/test_reorg.py::TestExtract::test_weights_invalid[negative] - Fai...
FAILED tests/test_reorg.py::TestExtract::test_weights_invalid[zero] - IndexEr...
FAILED tests/test_reorg.py::TestExtract::test_weights_invalid[nan] - IndexErr...
7 failed, 2 passed, 29 deselected in 0.13s

On this branch:

tests/test_reorg.py::TestExtract::test_weights PASSED
tests/test_reorg.py::TestExtract::test_weights_num_samples[stratified] PASSED
tests/test_reorg.py::TestExtract::test_weights_num_samples[multinomial] PASSED
tests/test_reorg.py::TestExtract::test_weights_unnormalized[stratified] PASSED
tests/test_reorg.py::TestExtract::test_weights_unnormalized[multinomial] PASSED
tests/test_reorg.py::TestExtract::test_weights_stratified_follow_weights PASSED
tests/test_reorg.py::TestExtract::test_weights_invalid[negative] PASSED
tests/test_reorg.py::TestExtract::test_weights_invalid[zero] PASSED
tests/test_reorg.py::TestExtract::test_weights_invalid[nan] PASSED
======================= 9 passed, 29 deselected in 0.03s =======================

Full suite: python -m pytest tests gives 259 passed. pre-commit run (mypy, ruff, pylint) passes on the changed files.

Stratified resampling always returned len(weights) draws and compared
the raw cumulative weights against uniforms in [0, 1), so unnormalized
weights collapsed the resample onto the first draws. Normalize the
weights once in extract, reject negative/non-finite/all-zero weights and
draw num_samples indices in _stratified_resample.

Fixes arviz-devs#236
@read-the-docs-community

Copy link
Copy Markdown

Documentation build overview

📚 arviz-base | 🛠️ Build #35560555 | 📁 Comparing 6aa8534 against latest (b87afe3)

  🔍 Preview build  

3 files changed
± _modules/arviz_base/reorg.html
± api/generated/arviz_base.dict_to_dataset.html
± api/generated/arviz_base.extract.html

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

extract with weights ignores num_samples and doesn't normalize the weights (stratified resampling)

1 participant