Expose forecast counts aligned to summarize_scores() grouping?

[annakrystalli]

Context

In hubEvals we've added an opt-in argument to our score_model_out() wrapper that appends a count column to summarised scores: the number of forecasts scored in each summary group. It's useful context (it shows how much data each summarised mean rests on, and surfaces sparse or uneven coverage across models/groups), and hubPredEvalsData (which prepares the per-model data behind the hubverse evaluation dashboards) needs it to report the number of predictions scored per model (hubverse-org/hubPredEvalsData#19).

The gap

scoringutils already counts forecasts via get_forecast_counts(), which operates on the forecast object and returns a count column (de-duplicating over quantile_level / sample_id). But summarize_scores() exposes no count aligned to its by grouping. So to attach "number scored per summary group" we count the per-row scores before summarize_scores() collapses them, then join it back:

counts <- dplyr::count(scores, dplyr::across(dplyr::all_of(by)))
summ   <- summarize_scores(scores, by = by)
summ   <- dplyr::left_join(summ, counts, by = by)  # + restore the `metrics` attribute

Question

Would you consider exposing a group count from summarize_scores() natively, e.g. an opt-in argument that also returns the group size alongside the summarised metrics? Or is there a recommended pattern we've missed? We appreciate that the design deliberately keeps counting on the forecast side (get_forecast_counts()), so we're really asking whether a summary-aligned count belongs in scoringutils or is better left to callers.

An extra reason it might belong upstream

Appending a count column onto a scores object downstream interacts awkwardly with get_forecast_unit(), which derives the forecast unit by elimination (any column that isn't protected or a listed metric is treated as a task-ID). A bolted-on count column is therefore misclassified as a forecast-unit column, which would skew the guards in get_pairwise_comparisons() / add_relative_skill() if such an object were routed back through them. If scoringutils owned the count column, it could protect the name in its class machinery and avoid this downstream footgun.

Reference implementation: hubverse-org/hubEvals#145.

[nickreich]

Adding a +1 to this being useful and important, as the count of scored units/tasks I think is an important and relevant metric that should be reported with summary scores!

[seabbs]

Thanks for this - nice and clear. It is a bit of a duplication but I see the utility. The implementation would also be straightforward. I don't currently see another way of doing this that keeps things modular (due to the count and then join path you flag here). I am minded to think we should implement this. I will ponder it briefly and then do so if nothing else occurs.