Skip to content

Fail loudly in mape and define smape's 0/0 term - #1050

Merged
jeongyoonlee merged 2 commits into
uber:masterfrom
arpitjain099:fix/mape-smape-degenerate-targets
Oct 3, 2026
Merged

jeongyoonlee merged 2 commits into
uber:masterfrom
arpitjain099:fix/mape-smape-degenerate-targets

Conversation

@arpitjain099

Copy link
Copy Markdown
Contributor

Fixes #1047.

mape filtered out near-zero targets and then averaged what was left. When every target is at or below EPS the filter selects nothing, and np.mean of an empty slice returns NaN with only a RuntimeWarning, which is easy to lose in training logs. The scalar sibling ape already asserts against this, so mape now raises ValueError.

smape has the related problem one element along, and it is the more likely one to bite: where target and prediction are both zero the term is 0/0, so a single such row turns the whole metric into NaN even when every other row is fine. That row is an exact prediction rather than an undefined one, so the term is defined as 0.

The issue was filed from static analysis, so I ran it first. Before:

mape(zeros, zeros)   -> nan   ['Mean of empty slice.', 'invalid value encountered in scalar divide']
smape(zeros, zeros)  -> nan
smape([0, 2, 0, 8], [0, 1.8, 0.5, 7])  -> nan

After: mape raises, smape(zeros, zeros) is 0.0, and the mixed case is 0.5596. Both metrics are unchanged on non-degenerate input (mape 0.10625 and smape 0.10727 before and after), and mape still ignores zero targets when others are present.

Five tests added to tests/test_metrics.py. Three fail on master, the two that pin unchanged behaviour pass either way.

mape filtered out near-zero targets and then took the mean of what was left.
With every target at or below EPS the filter selects nothing and np.mean of an
empty slice returns NaN with only a RuntimeWarning, while the scalar sibling
ape asserts against exactly this case. Raise instead.

smape had the related problem one step along: an element where target and
prediction are both zero gives 0/0, so a single such row turns the whole
metric into NaN even when every other row is fine. That case is an exact
prediction rather than an undefined one, so the term is defined as 0.

Signed-off-by: Arpit Jain <arpitjain099@gmail.com>
Signed-off-by: Arpit Jain <arpitjain099@gmail.com>
@jeongyoonlee
jeongyoonlee self-requested a review October 3, 2026 19:32

@jeongyoonlee jeongyoonlee left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

@jeongyoonlee
jeongyoonlee merged commit 3f032a9 into uber:master Oct 3, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

mape silently returns NaN when all targets are near-zero (empty filtered slice)

2 participants