Skip to content

fix(calibration): score the differentiable loss on the calibration period - #7

Merged
DarriEy merged 2 commits into
mainfrom
fix/calibration-period-leakage
Jul 28, 2026
Merged

fix(calibration): score the differentiable loss on the calibration period#7
DarriEy merged 2 commits into
mainfrom
fix/calibration-period-leakage

Conversation

@DarriEy

@DarriEy DarriEy commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Problem

kge_loss and nse_loss dropped warmup and then scored everything that remained. With a calibration/evaluation split configured, that span covers both windows.

Two consequences:

  1. Evaluation-period leakage. Gradient-based optimizers were trained on the held-out evaluation period, so the split-sample test was not honest.
  2. Mismatched reporting. The score the optimizer reported was computed over a different window than the Calib_* metrics written at final evaluation.

HEC-HMS had no completed ADAM run in the P3 ensemble, so this is fixed before it could produce misleading numbers. First run on fixed code: best 0.9010 == Calib_KGE 0.9010, with Eval_KGE 0.7946 genuinely held out.

Every other algorithm in the ensemble agreed exactly, because they score through _run_simulation rather than the JAX loss. L-BFGS was unaffected only because its configs request finite-difference gradients.

Fix

Thread a calibration slice through both losses, both backends, and the gradient-function factories. The worker supplies it from the shared InMemoryModelWorker.get_calibration_slice(), so all packages share one implementation. Requires the companion SYMFLUENCE PR.

Tests

New test_calibration_window.py (6 tests): a slice covering the whole post-warmup record is a no-op, a narrower slice changes the score, and two disjoint windows do not collapse to the same number.

These were mutation-tested — disabling the slice in _eval_window fails 4 of the 6 — so an accidental revert cannot pass silently.

Same defect found and fixed across jHBV, jTOPMODEL, jSAC-SMA, jXAJ, jHEC-HMS, jFUSE and cFUSE.

🤖 Generated with Claude Code

DarriEy and others added 2 commits July 28, 2026 13:02
…riod

kge_loss and nse_loss dropped warmup and then scored everything that
remained. With a calibration/evaluation split configured, that span covers
both windows, so gradient-based optimizers were trained on the held-out
evaluation period — the split-sample test was not honest — and the score
they reported was computed over a different window than the Calib_* metrics
written at final evaluation.

Same defect as jTOPMODEL, jSAC-SMA and jXAJ. Thread a calibration slice
through both losses, both backends, and the gradient-function factories;
the worker supplies it from the shared
InMemoryModelWorker.get_calibration_slice().

Verified on the P3 Bow-at-Banff ensemble: ADAM reports best 0.9010 ==
Calib_KGE 0.9010, with Eval_KGE 0.7946 genuinely held out.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Guards the leakage fix directly: a cal_slice covering the whole
post-warmup record must be a no-op, a narrower slice must change the
score, and two disjoint windows must not collapse to the same number.

Verified these fail against the pre-fix behaviour — disabling the slice
in _eval_window fails 4 of the 6 cases, so an accidental revert cannot
pass silently.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@DarriEy
DarriEy merged commit f7d847c into main Jul 28, 2026
2 checks passed
@DarriEy
DarriEy deleted the fix/calibration-period-leakage branch July 28, 2026 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant