Count every event in a time-series aggregation

[florian6973]

Stacked on #169 (agent/fix-value-feature-filter), which is not yet merged, so its commit appears in
this diff too. The change here is one line plus tests.

The defect

compute_agg collapses events to one row per (subject_id, time) before applying the rolling window.
The collapse is index-based: it hands aggregate_matrix the first and last row index of each group,
and aggregate_matrix slices matrix[min_index:max_index] and skips a window whose bounds are equal.

max_index is the inclusive index of the last event in the group, so the exclusive slice dropped
the last event of every timestamp, and dropped a timestamp holding a single event entirely.
get_rolling_window_indicies already adds the 1 for exactly this reason.

windows = group_df.select(pl.col("min_index", "max_index"))        # before
windows = group_df.select(pl.col("min_index"), pl.col("max_index") + 1)  # after

The loss is invisible in normal use: the matrix has the right shape, the run exits 0, and the only
symptom is that features are smaller than the data. On a dataset where every event has its own
timestamp it is total — every time-series aggregation comes out empty.

Minimal reproduction, code/count over columns A, B, C with a label after the last event:

events	before	after
A@t1, B@t2, C@t3	[0, 0, 0]	[1, 1, 1]
A@t1, B@t1, C@t2	[1, 0, 0]	[1, 1, 1]
A@t1, B@t1, C@t1	[1, 1, 0]	[1, 1, 1]

Measured effect on real data

MEDS-DEV MIMIC-IV demo extract, 100 subjects, tabularized at
aggs=[static/present,code/count,value/count,value/sum,value/sum_sqd,value/min,value/max],
window_sizes=[2h,12h,1d,7d,30d,365d,full], min_code_inclusion_count=10, labels from
mortality/in_icu/first_24h. Both arms share one environment, one dataset and one code-metadata file;
only this line differs.

The dataset holds 1,022,455 timestamped events in 96,849 (subject_id, time) groups, so 9.47% of
events were being dropped.

aggregation	matrices	before nnz	after nnz	delta	ratio
static/present	3	74	74	+0	1.000
code/count	21	222,302	237,251	+14,949	1.067
value/count	21	90,104	96,574	+6,470	1.072
value/sum	21	81,827	88,204	+6,377	1.078
value/sum_sqd	21	81,840	88,217	+6,377	1.078
value/min	21	5	4	−1	0.800
value/max	21	429	452	+23	1.054
total		476,581	510,776	+34,195	1.072

Total counted-event mass rises 10.63% for code/count (1,353,446 → 1,497,309), which is what the
drop rate predicts: recovering 9.47% of events raises a base of 90.53% by 1/0.9053 − 1 = 10.46%. Two
independent measurements agreeing to a fraction of a percent.

static/present is unchanged, as it is not windowed. value/min loses one stored entry, because
including a previously dropped event changed a minimum to zero, which is then not stored — the change
is not "every number goes up", it is "every event is seen".

Why the test suite did not catch it

tests/test_tabularize.py asserts matrix shapes and feature names, and that matrices are
non-empty; it never asserts matrix contents for a time-series aggregation. A matrix that is the
right shape and missing 9.5% of its mass passes every existing check. The added tests assert contents.

Tests

tests/test_generate_summarized_reps.py, 7 tests: code/count sees every event under three timestamp
layouts, value/sum equals the sum of the numeric values under the same three, and a lone event is not
dropped. All 7 fail on the parent commit and pass here.

The existing suite is unaffected: 81 passed before, 88 passed after (81 + 7 new), on
Python 3.12.13, linux/aarch64.

[florian6973]

Follow-up measurement, because "9.5% of events" understates this: the loss is systematic, not
uniform. The dropped event is the last row of its (subject_id, time) group in file order, and a
MEDS extract writes events in a deterministic per-source-table order — so a code that consistently
sorts last inside its timestamp group is not thinned, it is erased.

On the MIMIC-IV demo extract, per code, over the 1,474 codes with at least 50 events:

	codes
lose 100% of their events	2
lose more than half	118
lose nothing	107

Per-code drop rate: p10 = 0.8%, median 4.4%, p90 = 42.1%, max 100%. The worst-hit codes with
≥200 events:

dropped	code
100.0% (236/236)	TRANSFER_TO//ED//Emergency Department
99.8% (495/496)	SUBJECT_FLUID_OUTPUT//226560//ml
94.7% (820/866)	Blood Pressure
93.2% (750/805)	INFUSION_END//226452
89.5% (10,581/11,826)	SUBJECT_WEIGHT_AT_INFUSION//KG

So this is not 9.5% of noise spread evenly. It is Blood Pressure losing 19 of every 20 observations
and a transfer-to-ED indicator disappearing entirely, while other codes are untouched.

Does it change model performance?

It depends entirely on where the predictive code sits in its group — which is the point. A controlled
test, two synthetic cohorts of 600 subjects that differ only in the order of two events written at
the same timestamp, run through the full tabularize + XGBoost pipeline (the tutorial's 7 aggregations x
7 windows, 20 optuna trials, seeded):

cohort	before	after
predictive code written last at its timestamp (like TRANSFER_TO//ED)	AUROC 0.5232	AUROC 1.0000
predictive code written first at its timestamp	AUROC 1.0000	AUROC 1.0000

The first row is a perfectly learnable signal reduced to chance. The second is unaffected — and note
its code/count entries still rose from 6,307 to 10,507 with the fix, so a large loss of feature mass
and a loss of performance are not the same thing. Which one you get is decided by row order in the
extract, which no user of this library chooses or can see.