Skip to content

Add epoch-interval and H3 cell binning strategies - #2006

Merged
rfecher merged 11 commits into
masterfrom
motion/epoch-h3-binning
Nov 4, 2026
Merged

rfecher merged 11 commits into
masterfrom
motion/epoch-h3-binning

Conversation

@rfecher

@rfecher rfecher commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Statistics whose values are written directly per (time bucket, H3 cell), rather than derived from
ingested entries, need bins that are fixed-length, sort in time/cell order, and map a time window or
a geometry onto contiguous bin ranges. The existing TIME_RANGE strategy uses locale-dependent
bins with a leading flag byte, and SPATIAL bins carry per-strategy encodings that cannot be
queried as id ranges. This adds two strategies in
core/geotime/.../store/statistics/binning, meant to be combined as
CompositeBinningStrategy(EpochInterval(time), H3Cell(cell)):

  • EpochIntervalBinningStrategy (EPOCH_INTERVAL) with the public enum EpochInterval
    (HOUR, DAY, WEEK).
  • H3CellFieldValueBinningStrategy (H3_CELL).

Bin encodings

Both bins are exactly 8 bytes, Lexicoders.LONG.toByteArray(long), so they sort numerically.

  • Epoch: the bucket start in epoch milliseconds. HOUR and DAY align to the Unix epoch. WEEK is
    anchored at Monday 00:00 UTC (1970-01-05T00:00:00Z), independent of locale (unlike
    TemporalBinningStrategy's WEEK). Floor division, so pre-anchor and pre-1970 times bucket
    correctly, except GeoWave's null sentinel of -113 ms, which is treated as no time. A null or non-temporal value yields no bin (empty array from getBins).
  • H3: the cell id at the configured resolution (0-15). A cell at or finer than the resolution bins
    to cellToParent(cell, resolution); a coarser or invalid cell yields no bin. Field values may be
    Long/Number or a String H3 address.

Constraints

Strategy Constraint Result
Epoch Date, Calendar, Number, Instant the exact bin of that bucket
Epoch Interval exact bins for every bucket overlapping [start, end); an empty Interval (start == end, how a single instant is written) gives the bucket containing it
Epoch Interval of more than 100,000 buckets (MAX_EXACT_BUCKETS: about 11 years at HOUR) one range, no enumeration
Epoch unbounded Interval (endingAt, startingAt, ALL) one open range (null end on the unbounded side), no enumeration
Epoch Interval[] overlapping and adjacent windows merge; exact bins of the merged windows, or one range per merged window if any is unbounded or they cover more than 100,000 buckets in all
Epoch anything else parent handling (raw ByteArray, String, ranges, ...)
H3 Geometry, Geometry[], Envelope, Envelope[] bin ranges at resolution (H3CellRanges.geometryToRanges)
H3 Long, long[], Long[], String, String[], Collection<?> of those, cell at/finer than resolution exact bin of the cell's parent at resolution
H3 same, cell coarser than resolution descendant range (H3CellRanges.descendantRange)
H3 any range present everything returned as ranges; an exact bin b becomes [b, b]
H3 invalid cell id IllegalArgumentException

Helpers: Epoch has bucketStart(long), getBin(long), getInterval(ByteArray) and
ByteArrayRange[] getBinRanges(Interval): one contiguous range from the first to the last
overlapping bucket (usable directly for retention range deletes because the time sub-bin leads a
time-first composite key), a null end for an unbounded side, and no range at all for an empty
Interval, so BinConstraints.ofRange(hours.getBinRanges(window)) deletes nothing for an empty
window. Note the deliberate difference with constraints(), where an empty Interval selects the
bucket containing its instant; both entry points document this. A window's last instant is its end
minus one nanosecond, so a window whose end has sub-millisecond precision (from Instant.now())
still selects the bucket of its last instant. Times beyond ±(Long.MAX_VALUE / 2) ms, about
146 million years, are clamped to that limit when binning and constraining, so Instant.MIN and
Instant.MAX select the outermost buckets and a Date(Long.MIN_VALUE) value no longer wraps to a
far-future bin. H3 has getCell(ByteArray),
getBin(long) and getBinGeometry(ByteArray) (lon/lat polygon from H3CellRanges.cellToGeometry:
antimeridian cells are a MultiPolygon split at +/-180, polar cells are closed through the pole).
binToString gives the bucket as an ISO-8601 interval, or the H3 address.
Default tags: <field>-<INTERVAL> and <field>-H3(<resolution>).

Both strategies report getFixedBinLength() == 8, which composite prefixes, ranges and trailing
all-bins require of their children.

Both persist their own parameter (interval ordinal / resolution) ahead of the parent's fields.
Parameters: --binInterval and --binResolution. --binResolution is range-checked 0..15 by a
JCommander validator and again in fromBinary. Both strategies reject more than one --binField
(on use and in toBinary/fromBinary).

stat add checks that the binning strategy's options can be persisted before it adds the
statistic. Before, a second --binField on an empty type or with --skipCalculation printed a
stack trace, exited 0 and persisted nothing, because the statistics store logs and drops a
serialization failure. Now the command fails with "H3_CELL bins by exactly one field, but
--binField was given 2 times" (or EPOCH_INTERVAL) and a non-zero exit. The check is a few lines at
the top of AddStatCommand.computeResults, outside the new strategies, so it covers the REST path
too and any strategy whose serialization rejects its options. It cannot sit in prepare(), which
runs before the strategy's options are parsed, or in a JCommander validator, which sees one
occurrence at a time.

H3-leading composites: Composite(H3 spatial or H3 cell, X) constrained with {geometry, exactX}
throws IllegalArgumentException ("only 'all bins' may follow"), because a range child may be
followed only by all-bins. Use exact H3 bins instead (SpatialBinningType.H3.getSpatialBins(geometry, res), or cell ids at the strategy's resolution). Both cases are tested.

Time-leading composites: an unbounded Interval (startingAt, endingAt, ALL), or an
Interval[] that holds one, becomes a range. So in Composite(EpochInterval, X) it may be followed
only by allBins(). {Interval.startingAt(t), geometry}, {Interval.endingAt(t), cellId} and
{Interval.ALL, geometry} throw IllegalArgumentException ("only 'all bins' may follow such a
constraint"), while {Interval.startingAt(t), allBins()} gives one range. A bounded window of more
than 100,000 buckets becomes a range too, with the same rule. For an area query over all time, use
a bounded window of at most 100,000 buckets, or query the windows one at a time.

Registration

GeotimeRegisteredStatistics: EpochIntervalBinningStrategy = 2152, H3CellFieldValueBinningStrategy = 2153.

Dependencies (included commits)

Merge order: hbase-merge-delete-markers, statistics-store-fixes, composite-range-constraints,
h3-range-constraints, additive-vector, epoch-h3-binning, h3-motion-segmenter.
hbase-merge-delete-markers must land before statistics-store-fixes (see that PR for what to do
otherwise). This branch carries the earlier changes unmodified so its integration tests can run;
review only the last two commits:

  1. D: Keep delete markers through HBase server-side merging.
  2. G: Query binned statistics one at a time and honor range-only bin constraints.
  3. S: Batch statistic value writes and delete statistic values by bin range.
  4. C: Combine composite and multi-field bin constraints into prefixes and ranges (adds
    getFixedBinLength).
  5. H: Constrain H3 statistic bins with compact cell ranges (H3CellRanges, incl. cellToGeometry).

Once those merge, this rebases to just its own commits.

Testing

Unit tests in core/geotime (EpochIntervalBinningStrategyTest 28, H3CellFieldValueBinningStrategyTest 23, AddStatBinFieldTest 2):

  • bucket math (boundaries, Mondays, Sunday 23:59:59.999, pre-1970), bin ordering, getInterval
    and getBinRanges; empty and unbounded Intervals (endingAt, startingAt, ALL) through both
    getBinRanges and constraints;
  • every constraint shape; overlapping inputs come back as disjoint ranges;
  • H3 bins at, finer and coarser than the resolution for Long and String inputs, geometry ranges
    equal to the H3CellRanges cover, getBinGeometry (ordinary, antimeridian, both poles),
    binToString;
  • windows past MAX_EXACT_BUCKETS (the review's far-past and far-future windows, 1900 to 2026 at
    HOUR, exactly 100,000 against 100,001 buckets, an Interval[] and a composite led by one),
    Instant.MIN/MAX and windows at those limits, merged Interval[] windows and sub-millisecond
    ends;
  • fixed bin length, --binResolution validation (CLI and fromBinary), single --binField;
  • AddStatBinFieldTest: stat add through the CLI parser against a memory store from a config
    file, with two --binField values for each strategy (fails, persists nothing, with and without
    --skipCalculation) and with one (persists);
  • H3-leading composites: rejection with a clear message and the exact-bin workaround;
  • persistence round trips, and one in-memory DataStore round trip per strategy.

Verification: clean verify -pl core/store,core/geotime -am -DskipITs -Dformatter.action=validate
with SpotBugs at 0 (core/store 156 tests with 2 pre-existing skips, core/geotime 204, core/cli 56,
core/index 83, all green).

Integration tests (GeoWaveStatisticsIT, run on RocksDB and on HBase with server-side libraries,
9/9 each):

  • testTimeAndH3RangeConstraints: counts binned by Composite(EpochInterval DAY, H3 spatial) from
    the ingested grid, queried with day windows x areas (across the antimeridian, at the south pole,
    mid-latitudes) through getBinnedStatisticValues and aggregateStatistics, then a day-range
    delete.
  • testIncorporatedTimeAndH3Bins: counts written per (hour, H3 cell) around pentagons, the
    antimeridian, both poles and a city; random hour windows x areas; a key-range delete of a day, a
    prefix delete of one hour across all cells, an empty-window range delete (deletes nothing) and an
    open-ended Interval.endingAt(cutoff) retention delete; the remaining bins must match a
    brute-force model exactly.

Rich Fecher added 7 commits November 3, 2026 12:52
With the server-side library, MergingServerOp merges the cells of each
merging column family when HBase flushes a memstore or compacts store
files. It grouped cells by family, qualifier and tags only, so a delete
marker landed in the same group as a newer Put of the same column and
mergeList folded the two into one Put. The marker vanished, and the
older value it deleted in an earlier store file came back: write 5,
flush, delete, rewrite 3 and flush reads 8.

This hits every deletion of statistic values that is followed by a
rewrite before compaction: removeStatisticValue(s), recalcStatistic,
setStatisticValue and stat compaction (mergeStats). Delete cells now
pass through the merge unchanged.

While a scanner opened before a delete is still open, a flush or
compaction also keeps the older Put the marker masks and hands it to the
merge, which folded it into the rewrite with the same result. Puts that
a marker of the same row masks (at or below a DeleteColumn or
DeleteFamily, or at a version Delete or DeleteFamilyVersion) now pass
through unmerged as well. HBase hands a row over in batches of at most
hbase.hstore.compaction.kv.max cells, so the markers of a row are kept
in hints that the scanner wrapper shares across its batches.
RowMergingServerOp and the visibility variants inherit the fix.

GeoWaveHBaseStatisticDeleteIT covers an exact-bin and an all-bins delete
with flushes and a major compaction around the rewrite, a delete and
rewrite before any flush, and a delete with a scanner held open across
it before a flush, a major compaction, and a long row whose masked Put
comes in a later batch than the row delete.
With bin constraints, BaseDataStore.internalQueryStatistics passed every
matching statistic to each statistic's value query, so values were
duplicated and each statistic's constraints were applied to the others.
It also kept a statistic only when its constraints had exact bins or were
"all bins", so range-only constraints, which S2 and geohash geometry
constraints already produce, returned nothing from queryStatistics and
aggregateStatistics. Each statistic is now queried alone, and range-only
constraints are honored.

A constraint with both exact or prefix bins and ranges read only the
bins. StatisticsValueIterator now reads their union, each value once:
prefix bins become ranges, the ranges are merged, and exact bins that no
range covers are read on their own. Inverted ranges are dropped first,
since an empty range list means every value to a metadata query.

The in-memory store's range read also returned the entry at the range's
next prefix, one bin past an inclusive range end; its upper bound is now
exclusive.
Streaming writers flush many binned statistic values at once and expire
old time bins by range, so the statistics store needs batched writes and
range deletes.

- Add incorporateStatisticValues, which writes a map of bins to values
  with a single writer so stores can batch the writes.
- Add removeStatisticValues by BinConstraints (exact, prefix, range) to
  DataStore, backed by range and exact/prefix deletes in the statistics
  store. Open range ends map to the statistic's own value prefix and
  inverted ranges match nothing.
- removeStatisticValues(statistic) stopped after the first deleted bin;
  delete all bins with one prefix delete.
- Make metadata deleters honor primary id ranges and prefixes together
  with the secondary id in HBase, Accumulo, Cassandra and DynamoDB
  (Cassandra and DynamoDB previously dropped the table for a range), and
  report whether anything was deleted where that is cheap. The return
  values are best effort, as the javadoc now says for every overload.
- Accumulo compared a range start with the raw end, so an open or
  inverted range became a whole-table range; compare with the following
  prefix and drop rejected ranges. Data query ranges share the helper.
- RocksDB and FileSystem range reads returned and missed bins of other
  lengths, because a shorter bin's keys continue with its secondary id.
  Scan to the shortest prefix of the range end that can still be a bin
  at or after the start, and filter by primary id.
- Fix the Cassandra range reader ignoring the secondary id.
…anges

Composite and multi-field bins put their leading components first, so a
constraint on the leading components is a key prefix. The constraint
combining did not use that: exact bins on one component with 'all bins' on
the next threw, exact bins followed by ranges silently matched nothing,
'all bins' on every component matched nothing, and a prefix or range on the
last component hit an index error from a loop that incremented the wrong
variable.

CompositeBinningStrategy and FieldValueBinningStrategy now share one
combination in BinningStrategyUtils.combineConstraints. Exact leading
components multiply out. The first component that is not exact may be
'all bins', prefixes or ranges, which become prefixes or ranges under each
leading combination; every later component must be 'all bins'. Constraints
with every component exact keep the full bin encoding. Null range ends stay
within the leading bins rather than becoming unbounded, empty ranges are
dropped, and overlapping ranges are merged so stores that scan each range
separately do not read a bin twice.

Only constraints that can be answered exactly are accepted; the rest throw
IllegalArgumentException. Nothing separates one composite child bin from
the next, so StatisticBinningStrategy gains getFixedBinLength(), and a
composite needs a fixed length on every exact child before a prefix, range
or trailing 'all bins', and on the child constrained by prefix or range,
with constraint values that fit it. CompositeBinningStrategy (when every
child has one) and SpatialFieldValueBinningStrategy implement it.
Multi-field bins are delimited by their separator instead, so exact bins
alongside ranges become separator-terminated prefixes, and are refused on
the last field, where no range can stop exactly at the end of a value. This
also refuses again the {type, allBins} query on the DataType x Partition
composite, which would otherwise match every type sharing the name prefix.

Multi-field constraints are joined with a new overridable joinFieldBins, the
same method getBins uses, so SpatialFieldValueBinningStrategy, which
concatenates fixed-length field bins without a separator, now builds
constraints that match its bins; before, they were joined with '|' and
matched nothing. No bin encoding changes.

Also fixed:
- NumericRange and TimeRange field value strategies fell back to
  constraints() instead of singleFieldConstraints() for constraints they do
  not handle themselves, which recursed until a StackOverflowError for one
  field and threw for several, so a field could not be 'all bins'.
- CompositeBinningStrategy.getSubBinAtIndex skipped one sub-bin too few, so
  binMatches compared the wrong sub-bin for every child after the first.
- binMatches on a nested composite read the empty directly-assignable index
  and threw NoSuchElementException.
A geometry constraint on an H3-binned spatial statistic became one exact
bin lookup per cell of the geometry's polyfill, so query cost grew with
area at the statistic's resolution. The same cover is now compacted with
compactCells and each compact cell becomes the range of its descendants
at the target resolution, with ranges merged when no valid cell lies
between them. A 1 degree box at resolution 8 goes from 12,447 lookups to
293 ranges.

The ranges are exact. H3 ids never set bit 63, so the LONG lexicoded
bins sort in id order, and a cell's descendants at resolution R differ
from it (resolution field set to R) only in digits r+1..R, which run
0..6. Every valid resolution R id between the first and last descendant
is a descendant; the invalid ids in between are never stored as bins.

The cell math moves to a public H3CellRanges (shared H3Core, bin
encoding, descendant bounds and ranges, geometry to ranges, cell
geometry) for reuse by other H3 binning strategies, and H3BinningHelper
delegates to it. Two older defects that the new public methods expose
are fixed with it:

- H3 reads a polygon edge spanning more than 180 degrees of longitude as
  crossing the antimeridian, so world extents, polar caps over all
  longitudes and other wide envelopes were covered by their complement.
  Polygons wider than 90 degrees, or outside [-180, 180], are now cut
  into slabs of at most 90 degrees, shifted into [-180, 180], and the
  slab covers unioned. The overlay throws on invalid polygons, which
  would fail ingest writes, so an invalid polygon that needs slabs is
  first made valid with GeometryFixer, keeping collapsed line work.
  Empty geometries cover nothing instead of throwing from the centroid
  fallback.
- Bin geometries took the raw boundary longitudes, so cells on the
  antimeridian wrapped the globe and polar cells were invalid or missed
  their centers. cellToGeometry unwraps the longitudes, splits a cell
  crossing +/-180 into a multipolygon and closes a polar cell along the
  pole. A vertex exactly at a pole becomes two pole vertices at its
  neighbors' longitudes.

toRanges no longer copies a set input into a second boxed set.
Statistics whose values are written directly per (time bucket, H3
cell) need bins that are fixed length, sort in time and cell order,
and let a time window or a geometry map onto contiguous bin ranges.

EpochIntervalBinningStrategy bins a time field into fixed UTC hour,
day or week buckets (week anchored on Monday 00:00 UTC, independent
of locale). H3CellFieldValueBinningStrategy bins a field holding an
H3 cell id at a fixed resolution and turns geometry and coarse cell
constraints into id ranges via H3CellRanges. Both bins are the 8 byte
sortable Lexicoders.LONG encoding, and both report that fixed length,
so a time-first composite of the two takes exact buckets x cell
ranges. Registered as ids 2152 and 2153.

getBinRanges(Interval) gives the contiguous bucket range for a window,
with unbounded sides as open range ends and no range for an empty
window, so retention can range-delete directly. As a query constraint
an empty interval still selects the bucket of its instant. Both
strategies take exactly one field, and the H3 resolution is checked
on the command line and when read back.

Bounded windows of more than 100,000 buckets (MAX_EXACT_BUCKETS) are
constrained as ranges instead of enumerating every bucket, and
overlapping or adjacent windows of an Interval[] merge, so stores that
scan each range separately read no bucket twice. Times past
+/-(Long.MAX_VALUE / 2) ms fall in the bucket at that limit instead of
throwing or overflowing, and a window's last instant is its end minus a
nanosecond, so a sub-millisecond end still selects its bucket.

AddStatCommand checks that the binning strategy's options can be
persisted before adding the statistic: they are parsed only after
prepare(), and the statistics store logs and drops a serialization
failure, so a second --binField used to exit 0 and persist nothing.
Two integration tests for statistics binned by a time-first composite
with an H3 trailing bin:

- Counts binned by Composite(EpochInterval DAY, H3 spatial bins) from the
  ingested grid, queried with day windows x areas (across the
  antimeridian, at the south pole, mid-latitudes) through both
  getBinnedStatisticValues and aggregateStatistics, then a range delete of
  a span of days.
- Counts written directly per (hour, H3 cell) in two batched flushes
  around pentagons, the antimeridian, both poles and a city, queried with
  random hour windows x areas, then a key-range delete of a day, a prefix
  delete of one hour across all cells, an empty-window delete that must
  remove nothing and an open-ended retention delete before a cutoff; the
  remaining bins must match exactly.

Expected values are computed by brute force from the same polyfill cover.
Rich Fecher added 4 commits November 4, 2026 05:45
… motion/epoch-h3-binning

# Conflicts:
#	test/src/test/java/org/locationtech/geowave/test/GeoWaveITSuite.java
#	test/src/test/java/org/locationtech/geowave/test/basic/GeoWaveHBaseStatisticDeleteIT.java
@rfecher
rfecher marked this pull request as ready for review November 4, 2026 14:53
@rfecher
rfecher merged commit 15b7278 into master Nov 4, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant