Repository navigation
Add epoch-interval and H3 cell binning strategies - #2006
Merged
Merged
Conversation
added 7 commits
November 3, 2026 12:52
With the server-side library, MergingServerOp merges the cells of each merging column family when HBase flushes a memstore or compacts store files. It grouped cells by family, qualifier and tags only, so a delete marker landed in the same group as a newer Put of the same column and mergeList folded the two into one Put. The marker vanished, and the older value it deleted in an earlier store file came back: write 5, flush, delete, rewrite 3 and flush reads 8. This hits every deletion of statistic values that is followed by a rewrite before compaction: removeStatisticValue(s), recalcStatistic, setStatisticValue and stat compaction (mergeStats). Delete cells now pass through the merge unchanged. While a scanner opened before a delete is still open, a flush or compaction also keeps the older Put the marker masks and hands it to the merge, which folded it into the rewrite with the same result. Puts that a marker of the same row masks (at or below a DeleteColumn or DeleteFamily, or at a version Delete or DeleteFamilyVersion) now pass through unmerged as well. HBase hands a row over in batches of at most hbase.hstore.compaction.kv.max cells, so the markers of a row are kept in hints that the scanner wrapper shares across its batches. RowMergingServerOp and the visibility variants inherit the fix. GeoWaveHBaseStatisticDeleteIT covers an exact-bin and an all-bins delete with flushes and a major compaction around the rewrite, a delete and rewrite before any flush, and a delete with a scanner held open across it before a flush, a major compaction, and a long row whose masked Put comes in a later batch than the row delete.
With bin constraints, BaseDataStore.internalQueryStatistics passed every matching statistic to each statistic's value query, so values were duplicated and each statistic's constraints were applied to the others. It also kept a statistic only when its constraints had exact bins or were "all bins", so range-only constraints, which S2 and geohash geometry constraints already produce, returned nothing from queryStatistics and aggregateStatistics. Each statistic is now queried alone, and range-only constraints are honored. A constraint with both exact or prefix bins and ranges read only the bins. StatisticsValueIterator now reads their union, each value once: prefix bins become ranges, the ranges are merged, and exact bins that no range covers are read on their own. Inverted ranges are dropped first, since an empty range list means every value to a metadata query. The in-memory store's range read also returned the entry at the range's next prefix, one bin past an inclusive range end; its upper bound is now exclusive.
Streaming writers flush many binned statistic values at once and expire old time bins by range, so the statistics store needs batched writes and range deletes. - Add incorporateStatisticValues, which writes a map of bins to values with a single writer so stores can batch the writes. - Add removeStatisticValues by BinConstraints (exact, prefix, range) to DataStore, backed by range and exact/prefix deletes in the statistics store. Open range ends map to the statistic's own value prefix and inverted ranges match nothing. - removeStatisticValues(statistic) stopped after the first deleted bin; delete all bins with one prefix delete. - Make metadata deleters honor primary id ranges and prefixes together with the secondary id in HBase, Accumulo, Cassandra and DynamoDB (Cassandra and DynamoDB previously dropped the table for a range), and report whether anything was deleted where that is cheap. The return values are best effort, as the javadoc now says for every overload. - Accumulo compared a range start with the raw end, so an open or inverted range became a whole-table range; compare with the following prefix and drop rejected ranges. Data query ranges share the helper. - RocksDB and FileSystem range reads returned and missed bins of other lengths, because a shorter bin's keys continue with its secondary id. Scan to the shortest prefix of the range end that can still be a bin at or after the start, and filter by primary id. - Fix the Cassandra range reader ignoring the secondary id.
…anges
Composite and multi-field bins put their leading components first, so a
constraint on the leading components is a key prefix. The constraint
combining did not use that: exact bins on one component with 'all bins' on
the next threw, exact bins followed by ranges silently matched nothing,
'all bins' on every component matched nothing, and a prefix or range on the
last component hit an index error from a loop that incremented the wrong
variable.
CompositeBinningStrategy and FieldValueBinningStrategy now share one
combination in BinningStrategyUtils.combineConstraints. Exact leading
components multiply out. The first component that is not exact may be
'all bins', prefixes or ranges, which become prefixes or ranges under each
leading combination; every later component must be 'all bins'. Constraints
with every component exact keep the full bin encoding. Null range ends stay
within the leading bins rather than becoming unbounded, empty ranges are
dropped, and overlapping ranges are merged so stores that scan each range
separately do not read a bin twice.
Only constraints that can be answered exactly are accepted; the rest throw
IllegalArgumentException. Nothing separates one composite child bin from
the next, so StatisticBinningStrategy gains getFixedBinLength(), and a
composite needs a fixed length on every exact child before a prefix, range
or trailing 'all bins', and on the child constrained by prefix or range,
with constraint values that fit it. CompositeBinningStrategy (when every
child has one) and SpatialFieldValueBinningStrategy implement it.
Multi-field bins are delimited by their separator instead, so exact bins
alongside ranges become separator-terminated prefixes, and are refused on
the last field, where no range can stop exactly at the end of a value. This
also refuses again the {type, allBins} query on the DataType x Partition
composite, which would otherwise match every type sharing the name prefix.
Multi-field constraints are joined with a new overridable joinFieldBins, the
same method getBins uses, so SpatialFieldValueBinningStrategy, which
concatenates fixed-length field bins without a separator, now builds
constraints that match its bins; before, they were joined with '|' and
matched nothing. No bin encoding changes.
Also fixed:
- NumericRange and TimeRange field value strategies fell back to
constraints() instead of singleFieldConstraints() for constraints they do
not handle themselves, which recursed until a StackOverflowError for one
field and threw for several, so a field could not be 'all bins'.
- CompositeBinningStrategy.getSubBinAtIndex skipped one sub-bin too few, so
binMatches compared the wrong sub-bin for every child after the first.
- binMatches on a nested composite read the empty directly-assignable index
and threw NoSuchElementException.
A geometry constraint on an H3-binned spatial statistic became one exact bin lookup per cell of the geometry's polyfill, so query cost grew with area at the statistic's resolution. The same cover is now compacted with compactCells and each compact cell becomes the range of its descendants at the target resolution, with ranges merged when no valid cell lies between them. A 1 degree box at resolution 8 goes from 12,447 lookups to 293 ranges. The ranges are exact. H3 ids never set bit 63, so the LONG lexicoded bins sort in id order, and a cell's descendants at resolution R differ from it (resolution field set to R) only in digits r+1..R, which run 0..6. Every valid resolution R id between the first and last descendant is a descendant; the invalid ids in between are never stored as bins. The cell math moves to a public H3CellRanges (shared H3Core, bin encoding, descendant bounds and ranges, geometry to ranges, cell geometry) for reuse by other H3 binning strategies, and H3BinningHelper delegates to it. Two older defects that the new public methods expose are fixed with it: - H3 reads a polygon edge spanning more than 180 degrees of longitude as crossing the antimeridian, so world extents, polar caps over all longitudes and other wide envelopes were covered by their complement. Polygons wider than 90 degrees, or outside [-180, 180], are now cut into slabs of at most 90 degrees, shifted into [-180, 180], and the slab covers unioned. The overlay throws on invalid polygons, which would fail ingest writes, so an invalid polygon that needs slabs is first made valid with GeometryFixer, keeping collapsed line work. Empty geometries cover nothing instead of throwing from the centroid fallback. - Bin geometries took the raw boundary longitudes, so cells on the antimeridian wrapped the globe and polar cells were invalid or missed their centers. cellToGeometry unwraps the longitudes, splits a cell crossing +/-180 into a multipolygon and closes a polar cell along the pole. A vertex exactly at a pole becomes two pole vertices at its neighbors' longitudes. toRanges no longer copies a set input into a second boxed set.
Statistics whose values are written directly per (time bucket, H3 cell) need bins that are fixed length, sort in time and cell order, and let a time window or a geometry map onto contiguous bin ranges. EpochIntervalBinningStrategy bins a time field into fixed UTC hour, day or week buckets (week anchored on Monday 00:00 UTC, independent of locale). H3CellFieldValueBinningStrategy bins a field holding an H3 cell id at a fixed resolution and turns geometry and coarse cell constraints into id ranges via H3CellRanges. Both bins are the 8 byte sortable Lexicoders.LONG encoding, and both report that fixed length, so a time-first composite of the two takes exact buckets x cell ranges. Registered as ids 2152 and 2153. getBinRanges(Interval) gives the contiguous bucket range for a window, with unbounded sides as open range ends and no range for an empty window, so retention can range-delete directly. As a query constraint an empty interval still selects the bucket of its instant. Both strategies take exactly one field, and the H3 resolution is checked on the command line and when read back. Bounded windows of more than 100,000 buckets (MAX_EXACT_BUCKETS) are constrained as ranges instead of enumerating every bucket, and overlapping or adjacent windows of an Interval[] merge, so stores that scan each range separately read no bucket twice. Times past +/-(Long.MAX_VALUE / 2) ms fall in the bucket at that limit instead of throwing or overflowing, and a window's last instant is its end minus a nanosecond, so a sub-millisecond end still selects its bucket. AddStatCommand checks that the binning strategy's options can be persisted before adding the statistic: they are parsed only after prepare(), and the statistics store logs and drops a serialization failure, so a second --binField used to exit 0 and persist nothing.
Two integration tests for statistics binned by a time-first composite with an H3 trailing bin: - Counts binned by Composite(EpochInterval DAY, H3 spatial bins) from the ingested grid, queried with day windows x areas (across the antimeridian, at the south pole, mid-latitudes) through both getBinnedStatisticValues and aggregateStatistics, then a range delete of a span of days. - Counts written directly per (hour, H3 cell) in two batched flushes around pentagons, the antimeridian, both poles and a city, queried with random hour windows x areas, then a key-range delete of a day, a prefix delete of one hour across all cells, an empty-window delete that must remove nothing and an open-ended retention delete before a cutoff; the remaining bins must match exactly. Expected values are computed by brute force from the same polyfill cover.
added 4 commits
November 4, 2026 05:45
… motion/epoch-h3-binning # Conflicts: # test/src/test/java/org/locationtech/geowave/test/GeoWaveITSuite.java # test/src/test/java/org/locationtech/geowave/test/basic/GeoWaveHBaseStatisticDeleteIT.java
… motion/epoch-h3-binning
rfecher
marked this pull request as ready for review
November 4, 2026 14:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Statistics whose values are written directly per (time bucket, H3 cell), rather than derived from
ingested entries, need bins that are fixed-length, sort in time/cell order, and map a time window or
a geometry onto contiguous bin ranges. The existing
TIME_RANGEstrategy uses locale-dependentbins with a leading flag byte, and
SPATIALbins carry per-strategy encodings that cannot bequeried as id ranges. This adds two strategies in
core/geotime/.../store/statistics/binning, meant to be combined asCompositeBinningStrategy(EpochInterval(time), H3Cell(cell)):EpochIntervalBinningStrategy(EPOCH_INTERVAL) with the public enumEpochInterval(
HOUR,DAY,WEEK).H3CellFieldValueBinningStrategy(H3_CELL).Bin encodings
Both bins are exactly 8 bytes,
Lexicoders.LONG.toByteArray(long), so they sort numerically.anchored at Monday 00:00 UTC (1970-01-05T00:00:00Z), independent of locale (unlike
TemporalBinningStrategy's WEEK). Floor division, so pre-anchor and pre-1970 times bucketcorrectly, except GeoWave's null sentinel of -113 ms, which is treated as no time. A null or non-temporal value yields no bin (empty array from
getBins).resolution(0-15). A cell at or finer than the resolution binsto
cellToParent(cell, resolution); a coarser or invalid cell yields no bin. Field values may beLong/Numberor aStringH3 address.Constraints
Date,Calendar,Number,InstantIntervalIntervalof more than 100,000 buckets (MAX_EXACT_BUCKETS: about 11 years at HOUR)Interval(endingAt,startingAt,ALL)Interval[]ByteArray,String, ranges, ...)Geometry,Geometry[],Envelope,Envelope[]resolution(H3CellRanges.geometryToRanges)Long,long[],Long[],String,String[],Collection<?>of those, cell at/finer thanresolutionresolutionresolutionH3CellRanges.descendantRange)IllegalArgumentExceptionHelpers: Epoch has
bucketStart(long),getBin(long),getInterval(ByteArray)andByteArrayRange[] getBinRanges(Interval): one contiguous range from the first to the lastoverlapping bucket (usable directly for retention range deletes because the time sub-bin leads a
time-first composite key), a null end for an unbounded side, and no range at all for an empty
Interval, so
BinConstraints.ofRange(hours.getBinRanges(window))deletes nothing for an emptywindow. Note the deliberate difference with
constraints(), where an empty Interval selects thebucket containing its instant; both entry points document this. A window's last instant is its end
minus one nanosecond, so a window whose end has sub-millisecond precision (from
Instant.now())still selects the bucket of its last instant. Times beyond ±(
Long.MAX_VALUE / 2) ms, about146 million years, are clamped to that limit when binning and constraining, so
Instant.MINandInstant.MAXselect the outermost buckets and aDate(Long.MIN_VALUE)value no longer wraps to afar-future bin. H3 has
getCell(ByteArray),getBin(long)andgetBinGeometry(ByteArray)(lon/lat polygon fromH3CellRanges.cellToGeometry:antimeridian cells are a MultiPolygon split at +/-180, polar cells are closed through the pole).
binToStringgives the bucket as an ISO-8601 interval, or the H3 address.Default tags:
<field>-<INTERVAL>and<field>-H3(<resolution>).Both strategies report
getFixedBinLength() == 8, which composite prefixes, ranges and trailingall-bins require of their children.
Both persist their own parameter (interval ordinal / resolution) ahead of the parent's fields.
Parameters:
--binIntervaland--binResolution.--binResolutionis range-checked 0..15 by aJCommander validator and again in
fromBinary. Both strategies reject more than one--binField(on use and in
toBinary/fromBinary).stat addchecks that the binning strategy's options can be persisted before it adds thestatistic. Before, a second
--binFieldon an empty type or with--skipCalculationprinted astack trace, exited 0 and persisted nothing, because the statistics store logs and drops a
serialization failure. Now the command fails with "H3_CELL bins by exactly one field, but
--binField was given 2 times" (or EPOCH_INTERVAL) and a non-zero exit. The check is a few lines at
the top of
AddStatCommand.computeResults, outside the new strategies, so it covers the REST pathtoo and any strategy whose serialization rejects its options. It cannot sit in
prepare(), whichruns before the strategy's options are parsed, or in a JCommander validator, which sees one
occurrence at a time.
H3-leading composites:
Composite(H3 spatial or H3 cell, X)constrained with{geometry, exactX}throws
IllegalArgumentException("only 'all bins' may follow"), because a range child may befollowed only by all-bins. Use exact H3 bins instead (
SpatialBinningType.H3.getSpatialBins(geometry, res), or cell ids at the strategy's resolution). Both cases are tested.Time-leading composites: an unbounded Interval (
startingAt,endingAt,ALL), or anInterval[]that holds one, becomes a range. So inComposite(EpochInterval, X)it may be followedonly by
allBins().{Interval.startingAt(t), geometry},{Interval.endingAt(t), cellId}and{Interval.ALL, geometry}throwIllegalArgumentException("only 'all bins' may follow such aconstraint"), while
{Interval.startingAt(t), allBins()}gives one range. A bounded window of morethan 100,000 buckets becomes a range too, with the same rule. For an area query over all time, use
a bounded window of at most 100,000 buckets, or query the windows one at a time.
Registration
GeotimeRegisteredStatistics:EpochIntervalBinningStrategy= 2152,H3CellFieldValueBinningStrategy= 2153.Dependencies (included commits)
Merge order: hbase-merge-delete-markers, statistics-store-fixes, composite-range-constraints,
h3-range-constraints, additive-vector, epoch-h3-binning, h3-motion-segmenter.
hbase-merge-delete-markers must land before statistics-store-fixes (see that PR for what to do
otherwise). This branch carries the earlier changes unmodified so its integration tests can run;
review only the last two commits:
getFixedBinLength).H3CellRanges, incl.cellToGeometry).Once those merge, this rebases to just its own commits.
Testing
Unit tests in
core/geotime(EpochIntervalBinningStrategyTest28,H3CellFieldValueBinningStrategyTest23,AddStatBinFieldTest2):getIntervaland
getBinRanges; empty and unbounded Intervals (endingAt,startingAt,ALL) through bothgetBinRangesandconstraints;equal to the
H3CellRangescover,getBinGeometry(ordinary, antimeridian, both poles),binToString;MAX_EXACT_BUCKETS(the review's far-past and far-future windows, 1900 to 2026 atHOUR, exactly 100,000 against 100,001 buckets, an
Interval[]and a composite led by one),Instant.MIN/MAXand windows at those limits, mergedInterval[]windows and sub-millisecondends;
--binResolutionvalidation (CLI andfromBinary), single--binField;AddStatBinFieldTest:stat addthrough the CLI parser against a memory store from a configfile, with two
--binFieldvalues for each strategy (fails, persists nothing, with and without--skipCalculation) and with one (persists);Verification:
clean verify -pl core/store,core/geotime -am -DskipITs -Dformatter.action=validatewith SpotBugs at 0 (core/store 156 tests with 2 pre-existing skips, core/geotime 204, core/cli 56,
core/index 83, all green).
Integration tests (
GeoWaveStatisticsIT, run on RocksDB and on HBase with server-side libraries,9/9 each):
testTimeAndH3RangeConstraints: counts binned byComposite(EpochInterval DAY, H3 spatial)fromthe ingested grid, queried with day windows x areas (across the antimeridian, at the south pole,
mid-latitudes) through
getBinnedStatisticValuesandaggregateStatistics, then a day-rangedelete.
testIncorporatedTimeAndH3Bins: counts written per (hour, H3 cell) around pentagons, theantimeridian, both poles and a city; random hour windows x areas; a key-range delete of a day, a
prefix delete of one hour across all cells, an empty-window range delete (deletes nothing) and an
open-ended
Interval.endingAt(cutoff)retention delete; the remaining bins must match abrute-force model exactly.