Skip to content

Compute window std dev, skew and kurtosis together - #558

Merged
gbeane merged 2 commits into
mainfrom
perf/shared-window-moments
Nov 4, 2026
Merged

gbeane merged 2 commits into
mainfrom
perf/shared-window-moments

Conversation

@gbeane

@gbeane gbeane commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

Window features are now the largest share of feature compute time (~59% in a recent jabs-init --timing-report run). Profiling one identity showed the std_dev, skew, and kurtosis window operations together taking ~28% of window time, and each independently recomputes the same per-window count, mean, centered values, and variance for every column.

  • window_stats.window_std_skew_kurtosis(values, window) computes all three from one set of shared terms (_central_moment_terms). np_skew and np_kurtosis now use the same helpers, so the arithmetic is shared rather than duplicated. The standard deviation is sqrt of the same biased variance np.nanstd computes.
  • Feature._window_standard computes the three through it in one pass (_shared_moment_features), keeping the original operation order. An operation is only routed through the shared path if it still uses its default implementation, so a subclass that overrides one of them keeps its own.
  • The public window_std_dev / window_skew / window_kurtosis functions are unchanged.

Performance

tests/data/sample_pose_est_v6.h5, get_window_features(10) for all 4 identities, best of 3:

time
before 2.388 s
after 2.023 s (-15%)

Feature values

Unchanged — no FEATURE_VERSION bump. Verified by dumping every window feature (54,936 arrays) for every identity in the v5 and v6 sample pose files at window sizes 5 and 10 on main and on this branch, and comparing for exact equality including key order.

Testing

  • window_std_skew_kurtosis is bit-identical to the three separate ops on the existing test series (scattered nan, nan runs wider than the window, constant, all-nan, mostly zero) at three window sizes.
  • Feature.window results equal each window op run on its own, in op order; a subclass overriding skew keeps its implementation.
  • uv run pytest --ignore=tests/ui/test_viewer_window.py: 2090 passed, 7 skipped. tests/ui/test_viewer_window.py aborts the run with SIGTRAP after test_video_without_pose_file_opens_without_overlays locally on main too, so it's unrelated.

@gbeane
gbeane requested a balanced review from Copilot November 3, 2026 21:15
@gbeane gbeane self-assigned this Nov 3, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The shared reduction changes standard-deviation results for windows containing infinities from NaN to zero or infinity, altering feature values and requiring a fix with infinity-case equivalence tests.

1 open issue
What changed in this PR

This PR reduces window-feature computation time by sharing intermediate calculations for standard deviation, skew, and kurtosis in JABS feature extraction.

Changes:

  • Adds a combined window-statistics function and reusable central-moment helpers.
  • Routes default moment operations through the shared calculation while preserving subclass overrides and output order.
  • Adds equivalence tests for combined statistics and feature-level integration.
File Description
tests/​feature_extraction/​window_operations/​test_window_stats.py Tests combined statistics against separate operations.
tests/​feature_extraction/​test_feature_base_class.py Tests output equivalence, operation order, and subclass overrides.
src/​jabs/​feature_extraction/​window_operations/​window_stats.py Adds shared moment calculations and the combined statistics function.
src/​jabs/​feature_extraction/​feature_base_class.py Integrates shared calculations into standard window features.

Review effort: Balanced

Comment on lines +152 to +153
# equal to np.nanstd: both are the square root of the same biased variance
std_dev = np.sqrt(variance)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — fixed in 02c2b76. The std dev now sums the squared deviations like np.nanstd: only values that were nan in the input are zeroed, so a nan from inf - inf still propagates (skew/kurtosis keep their original nansum behavior, which was already shared). Added all-inf, mixed ±inf, and inf+nan series to the equivalence test at three window sizes (the previous commit fails all 9 cases); the 54,936-array comparison against main on the sample pose files is still exact.

@gbeane
gbeane merged commit 4f263ee into main Nov 4, 2026
5 checks passed
@gbeane
gbeane deleted the perf/shared-window-moments branch November 4, 2026 00:11
@gbeane gbeane mentioned this pull request Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants