Move time series management to InfraStore.jl - #608
Merged
Merged
Conversation
…store Wire InfrastructureSystems to delegate SingleTimeSeries data + metadata to the external Rust time-series-store engine behind `backend=:rust`. Forecast types remain on the legacy HDF5 path. - New src/rust_time_series_store.jl: RustTimeSeriesStore <: TimeSeriesStorage, a self-contained ccall wrapper (no dependency on the registered TimeSeries package). Data identity is the array content hash, not a UUID. - Route SingleTimeSeries through the real public API, guarded by `data_store isa RustTimeSeriesStore` (legacy path untouched): add_time_series!, get_time_series (full + start_time/len slice), has_time_series, remove_time_series!, get_time_series_counts, clear. - SystemData/TimeSeriesManager gain a `backend` selector. - serialize/deserialize(SystemData) write/read a .nc + standalone .sqlite pair (time_series_storage_type="RustTimeSeriesStore"); no HDF5, no embedded blob. - Tests: in-memory public-API round-trip (15) + System save/reload (8) + direct-store round-trip (17) + on-disk persistence (4). KNOWN: the on-disk path needs HDF5.jl to share the Rust dylib's libhdf5 (one libhdf5 per process); configure via LocalPreferences.toml (not committed). The hdf5_conflict_probe.jl test documents the conflict empirically. This goes away once all time series types move to Rust and IS drops HDF5. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…store - Add forecast ccall wrappers (add_forecast!, get_forecast_metadata, has_typed, remove_typed) and window flatten/reshape logic in rust_time_series_store.jl. - Route add/get/has/remove for AbstractDeterministic through the manager/public API; reads reconstruct the STORED type (Deterministic windows, or a DST wrapping the reconstructed underlying SingleTimeSeries). - Fix get_num_time_series/isempty to count forecasts too (otherwise a forecast-only System serialized as empty). - Tests: Deterministic + DST in-memory round-trip and System save/reload. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Add Probabilistic ccall wrappers (add_probabilistic!, get_probabilistic_metadata returning percentiles) and 3-D window flatten/reshape in rust_time_series_store.jl. - Widen forecast routing predicates from AbstractDeterministic to Forecast so Probabilistic is handled by add/get/has/remove; reconstruct the stored type. - Test: Probabilistic in-memory round-trip + System save/reload. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes Rust-backed support for all five time series types. Scenarios reuses the generic forecast FFI (ts_store_add_forecast / ts_store_get_forecast_metadata, TimeSeriesType=Scenarios) with no ts-store change: scenario_count is derived on read as length / (horizon_count * count). Stores the flattened 3-D (scenario_count, horizon_count, count) array by content hash. - Add Scenarios branches to add/get/has/remove routing (predicates already widened to Forecast); reconstruct the stored type. - Test: Scenarios in-memory round-trip + System save/reload. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
All five time series types now run on the Rust backend, so the legacy HDF5 storage is dead code. Removing it eliminates the in-process libhdf5 conflict permanently: IS.jl no longer loads HDF5, so the Rust on-disk NetCDF path works with no LocalPreferences/libhdf5-sharing workaround. - Delete src/hdf5_time_series_storage.jl (the Hdf5TimeSeriesStorage struct and all its methods) and remove HDF5 + H5Zblosc from Project.toml. - Remove the HDF5 SQLite-blob hack (to_h5_file / from_h5_file) and the InMemory→HDF5 conversion (convert_to_hdf5, the from-Hdf5 constructor). - make_time_series_storage now returns InMemoryTimeSeriesStorage (the only pure-Julia backend); on-disk persistence is provided exclusively by the Rust backend (`time_series_backend = :rust`). Add a generic no-op open_store!. - System serialize/deserialize: drop the HDF5 paths; non-empty non-Rust stores and legacy .h5 systems now raise a clear error directing to the Rust backend. - Remove the obsolete hdf5_conflict_probe test and the stale libhdf5 notes. In-memory and all Rust backend tests pass; HDF5 is no longer loaded. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Remove HDF5 from the test harness (`import HDF5`) and test/Project.toml; Aqua stale-deps/compat checks now pass with HDF5 gone. - Rewrite test_time_series_storage.jl to cover the in-memory backend only; drop the HDF5-specific data-format-version and on-disk compression testsets (NetCDF + SQLite persistence is covered by the Rust integration tests). - Remove dead HDF5 tests from test_time_series.jl (copy_h5_file, deepcopy-on-HDF5) and fix the storage-type assertion (InMemoryTimeSeriesStorage). - Move the standalone Rust integration tests to test/rust/ so the ReTest harness (which auto-includes test_*.jl and has no cdylib) doesn't pull them in; run them directly, e.g. `julia --project=. test/rust/rust_time_series_store.jl`. Harness loads cleanly; storage + in-memory tests pass. Disk-serialization round-trip tests still require the Rust backend (cdylib) and remain to migrate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`rust_time_series_store.jl` now delegates all low-level calls to the `TimeSeriesStore` binding package (the renamed standalone Julia binding); it keeps only the IS-specific glue: owner/feature conversion, the SingleTimeSeries and forecast window flatten/reshape, and the TimeSeriesManager routing. `RustTimeSeriesStore` wraps a `TimeSeriesStore.Store`. `RustTimeSeriesNotFound` is now an alias for `TimeSeriesStore.NotFoundError`. Adds TimeSeriesStore to [deps]/[compat] and `import TimeSeriesStore` to the module. Until the package is registered it must be `Pkg.develop`ed locally (Manifest is gitignored). All five Rust integration tests pass unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
System time series serialization now goes through the Rust backend (.nc arrays + standalone .sqlite metadata next to the JSON; no HDF5). Brings the existing serialization tests onto it: - validate_serialization moves the recorded storage file(s) — the Rust backend's .nc plus its sibling .sqlite — rather than the old single .h5. - The three time-series serialization testsets build :rust-backed systems, gated on the cdylib/JLL being available (rust_ts_available()); they @test_skip when it is absent (e.g. CI without the binary). - compare_values(::RustTimeSeriesStore, ::RustTimeSeriesStore) compares by counts (handle/path differ across a round-trip; element equality is covered by the Rust integration tests). With the cdylib present: system-data (2), read-only (3), mutable (2) all pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
serialize_single! keeps the value element type (Float64, Int64, …) instead of
forcing Vector{Float64}, and passes string(eltype) as the logical-type tag.
get_single reconstructs with the stored dtype (get_metadata now returns it;
get_array_by_hash takes the element type). An Int64 SingleTimeSeries now
round-trips through the Rust backend with its eltype intact; the Float64 path is
unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`SingleTimeSeries{T}` carries the value element type T (inferred from the data,
so callers never spell it out); the `data` field stays an untyped TimeArray.
This enables the parametric API:
ts = SingleTimeSeries(name="active_power", data=[1.2, 3.4, 5.6]) # {Float64}
add_time_series!(sys, component, ts)
ts2 = get_time_series(SingleTimeSeries{Float64}, component, "active_power")
- get_time_series(::Type{T}) where T<:TimeSeriesData already dispatches on the
concrete `SingleTimeSeries{Float64}`; the Rust get reconstructs with the
requested element type (falling back to the stored dtype).
- Relaxed `time_series_data_to_metadata` and `check_consistency` to
`::Type{<:SingleTimeSeries}` so the concrete parametric type dispatches.
- Added `get_data_type(::SingleTimeSeries{T}) = string(T)` (fixes the previously
undefined-symbol FunctionData cost tests).
SingleTimeSeries structs aren't serialized to JSON (only metadata + data are), so
serialization is unaffected. Validated: the parametric example end-to-end on the
Rust backend; Int64 + Float64 + Linear/Quadratic/PiecewiseLinear FunctionData
element types; serialization, transforms, consistency, and rust integration tests
all pass. (Pre-existing unrelated failure: bulk-add's in_memory=false expectation,
HDF5-removal fallout.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_rust_add_forecast! passes each forecast's native array — Deterministic (horizon_count, count), Probabilistic/Scenarios 3-D — instead of flattening to 1-D. _rust_get_forecast reconstructs via TimeSeriesStore.get_array_nd with dims derived from the forecast parameters (horizon÷resolution, count, percentiles), keeping the per-window extraction unchanged. Removed the now-unused _flatten_deterministic helper. All four forecast round-trip + system integration tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`time_series_in_memory=false` no longer selects an on-disk backend (HDF5 is gone; on-disk persistence is `time_series_backend=:rust`), so the non-Rust store is always in-memory. Assert `stores_time_series_in_memory(sys)` is true instead of `== in_memory`. Both bulk-add testsets pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixed-size FunctionData element types now round-trip through the Rust store:
serialize_single! encodes a Vector{LinearFunctionData}/Vector{QuadraticFunctionData}
as a (length, 2)/(length, 3) Float64 array tagged with its logical type; reads
key on the stored logical_type (from get_metadata) to rebuild the FunctionData,
so the non-parametric get_time_series(SingleTimeSeries, ...) returns the right
type. Scalar dtypes (Float64, Int64, …) are unchanged. PiecewiseLinearData
(ragged) raises a clear "not supported yet" error.
Added a rust test covering Int64, Quadratic (non-parametric get), and Linear
(parametric get); FunctionData also verified to persist to .nc/.sqlite and reload
with its element type. Removed the now-unused _rust_sts_eltype helper.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Each step has a variable number of (x, y) points, so it's stored as a padded (len, 1 + 2*max_points) Float64 matrix: column 1 of each row is the point count, keeping shape[0] = the timestep count. Reads derive the per-step width from the array size and rebuild each PiecewiseLinearData from its row. Completes FunctionData support on the Rust backend (Linear/Quadratic/PiecewiseLinear). Rust test extended with a ragged 2/3/2-point series. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…gleTimeSeries
Fixes surfaced by a full suite run with the cdylib present:
Parametric SingleTimeSeries{T} fallout (it's now a UnionAll, not a DataType):
- TimeSeriesFileMetadata.time_series_type and the parser assignment Set use Type
(not DataType); _get_all_concrete_subtypes records parametric leaf types via
their UnionAll body; add_serialization_metadata! skips .parameters for UnionAll;
added a SingleTimeSeries{T}(metadata, data) forwarding constructor for the
deserialize path.
Name-less has_time_series routing: has_time_series(owner) and
has_time_series(owner, T) now route to the Rust store (via has_for_owner /
_rust_has_any) instead of the dummy metadata store; previously returned false.
HDF5-removal fallout: bulk-add + compression assertions updated (in-memory is the
only non-Rust backend); supplemental-attribute + deserialized-system serialization
tests gated on the Rust backend (the latter skipped pending
scaling_factor_multiplier support); the custom-directory test uses the Rust
backend's .nc path; removed the obsolete transform/retransform test file (the
HDF retransform helper was removed).
Pre-existing in-memory bug: forecast row-slicing linearized matrix-valued windows
(Probabilistic/Scenarios); fixed with selectdim.
Suite: 8027 passed, 1 skipped, 0 failed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Rust backend already honors a custom directory for its .nc/.sqlite files (via the `time_series_directory` kwarg, the SIENNA_TIME_SERIES_DIRECTORY env var, or tempdir()). On HPC a per-job scratch directory may not exist yet, so mkpath it before creating the store instead of failing with "unable to open database file". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The TimeSeriesStore.jl binding consolidated its forecast API and moved the association `name` / `scaling_factor_multiplier` onto the time series structs. Update the Rust-backed store glue accordingly: - SingleTimeSeries: pass `name` (and scaling, logical_type) on the struct constructor; drop them from add_time_series!. - Forecast writes: build TSS.Deterministic / TSS.Probabilistic / TSS.Scenarios and route through the generic add_time_series! instead of the removed add_forecast! / add_probabilistic! transports. - DeterministicSingleTimeSeries: the binding derives it from a stored SingleTimeSeries via transform_single_time_series! rather than a direct forecast write, so persist the underlying series (if absent) and transform. - Forecast reads: use get_time_series(T, store, owner, name; ...) (returns the decoded array + params) in place of the removed get_*_metadata + get_array_nd. All standalone Rust integration tests pass (test/rust/*.jl, 129 assertions). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Plumb CompressionSettings from SystemData/TimeSeriesManager into RustTimeSeriesStore so the time-series-store NetCDF backend honors the requested compression. DEFLATE maps to the backend's level/shuffle; disabled compression maps to no filter; BLOSC raises an explicit error (unsupported by the Rust backend). open_rust_store now queries the persisted policy back over the FFI (get_compression) so get_compression_settings reports the real settings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove the SQLite TimeSeriesMetadataStore and InMemoryTimeSeriesStorage and make RustTimeSeriesStore the sole time-series backend (clean break: no time_series_backend kwarg, no metadata_store field). - src/rust_time_series_store.jl: full parity glue derived from TSS.list_metadata (metadata reconstruction incl. scaling_factor_multiplier, keys, multiple, resolutions, counts-by-type, summary tables, forecast parameters, owner-uuid listing, consistency check, replace_component_uuid!, per-owner clear). time_series_uuid is derived from the content hash. - time_series_manager.jl: struct is (data_store, read_only); Rust-only add/clear/remove/list; SingleTimeSeries-attached-to-DST removal guard. - system_data.jl: Rust-only serialize/deserialize; getters routed to glue; transform_single_time_series! uses the store-level transform. - time_series_interface.jl / component.jl / time_series_storage.jl / deterministic_single_time_series.jl: drop legacy paths and dead code. - tests: default to Rust; delete legacy-only tests (storage, SQLite migrations, optimize_database!, to_dataframe) and redundant test/rust POCs. Status: full suite ~green; one serialization round-trip test failing (double round-trip of a deserialized system). See HANDOFF_drop_legacy_backend.md for the exact remaining work and run steps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bring the Rust-only backend to feature parity with the removed legacy backend and get the full test suite passing (7952 pass, 0 fail/error, 3 documented @test_broken for store-model gaps). IS glue (rust_time_series_store.jl): - Partial (subset) feature/resolution matching for get/has/remove. - Forecast start_time/count/len window slicing + validation; Deterministic query also matches DeterministicSingleTimeSeries. - scaling_factor_multiplier threaded through add/read; content-hash time_series_uuid assigned on add. - FunctionData-valued Deterministic forecasts (encode/decode via logical_type). - Distinct-array counts, forecast-parameter compatibility checks, hash-based STS-attached-to-DST removal guard, begin_time_series_update rollback, component-only / resolution-filtered transforms. Tests: re-enabled the deserialized-system round-trip; marked two store-model gaps @test_broken with TODOs — irregular (Month/Year) resolutions and multiple-interval forecasts (the store key omits interval). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Design change: scaling_factor_multiplier is no longer supported. Remove the field from all time series types (SingleTimeSeries, Probabilistic, Scenarios, Deterministic) and their metadata descriptors (regenerated), along with the get_/set_ accessors. Also remove the now-meaningless read-time plumbing: the ignore_scaling_factors keyword across the public read API and cache, the scaling_factor_multiplier_mapping keyword in copy_time_series!, and the multiplier application in _make_time_array. Drop the parser fields and stop passing/reading the attribute at the Rust store boundary. Tests and docs updated; full suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Port the integer-id identity model so components and supplemental attributes are
identified by small integer ids assigned by SystemData when attached, instead of
UUIDs, while keeping the Rust time-series backend. Mirrors the IS2.jl model.
- InfrastructureSystemsInternal gains id::Int (UNASSIGNED_ID until attached);
identity goes through get_id/set_id!. Time series keep their UUIDs.
- SystemData tracks two independent id streams (next_component_id,
next_supplemental_attribute_id, each from 1); component_uuids -> component_ids,
subsystems -> Set{Int}; ComponentUUIDs -> ComponentIDs. ids are preserved
across serialization (assign_id! advances the counter past restored ids).
- Supplemental attribute associations use integer component_id/attribute_id.
- Rust time-series glue threads (owner_id, owner_category) to the category-aware
TimeSeriesStore.jl binding; assign_new_id! re-keys time series via replace_owner!.
- Tests ported; forecast tests fixed to use strictly-increasing percentiles
(the store validates this).
Full InfrastructureSystems.jl test suite passes against the Rust backend.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bring the integer-id model to parity with PR #587 (the pure-Julia integer-id PR), porting the review-driven fixes that were missing here: - assign_new_id_internal!: also remap the component's id in subsystem membership sets (previously left stale, breaking subsystem lookups after reassignment). - Guard against UNASSIGNED_ID on the manager-direct path when attaching a supplemental attribute and when adding an association, with actionable errors instead of colliding at id 0. - Fix "subystem" typos in system_data.jl and subsystems.jl. - Tests: assert subsystem membership tracks the new id after assign_new_id!; assign ids in the manager-direct supplemental attribute tests. (PR #587's f896870 put owner_category in the Julia metadata-store unique index; here that lives in the Rust store instead — NatLabRockies/infrastore#4.) Full test suite passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Route AbstractDeterministic reads through the family-resolving TimeSeriesStore API, and delegate array reference-counting to the core (count_array_references) instead of scanning metadata in Julia. - Remove the vestigial time-series UUID end to end: drop time_series_uuid from structs.json and the 4 generated metadata structs, the public get_time_series_uuid API and exports, and the _rust_ts_uuid / _rust_assign_stored_uuid! shim with all set_uuid! / _metadata_from_row sites. The Rust store is content-addressed; address a time series via TimeSeriesKey (the old UUID was hash-derived and not unique per association, so it was unsuitable as a handle). - docs: rewrite the stale HDF5 storage docs to describe the Rust NetCDF + SQLite content-addressed store; correct the supplemental attribute schema to integer IDs; refresh a stale design comment. Validated: IS.jl precompiles and the time-series test suite passes (4733 assertions, 0 failures, 3 pre-existing broken). BREAKING: removes the public get_time_series_uuid API and the time_series_uuid serialization field; needs PowerSystems.jl coordination. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The metadata structs (SingleTimeSeriesMetadata, DeterministicMetadata, ProbabilisticMetadata, ScenariosMetadata) carried the same fields as the TimeSeriesKey hierarchy plus a vestigial `internal`. With the Rust catalog owning the persistent record, they were a redundant transient view. Make TimeSeriesKey (StaticTimeSeriesKey / ForecastKey) the single descriptor. - Move the accessors and abstract-level methods onto the key hierarchy: get_initial_timestamp, get_features, get_length, get_horizon, get_interval, get_count, get_horizon_count, Base.length, and the static defaults (get_interval = nothing, get_count = 1). - Rename get_time_series_metadata(T, owner, name) -> get_time_series_key (returns a TimeSeriesKey); fold the owner-enumeration overload into get_time_series_keys(owner; filters...). Update callers (cache, copy, remove). - Build keys directly from catalog rows (_key_from_row); drop make_time_series_key. - Delete the metadata type system: the 4 generated structs and their structs.json entries, the abstract TimeSeriesMetadata/StaticTimeSeriesMetadata/ForecastMetadata types, the dead helpers (is_time_series_sub_type, time_series_data_to_metadata, time_series_metadata_to_data, the unused _get_columns/_get_rows/_check_start_time cluster), and the legacy T(metadata, data) reconstruction constructors. - Relocate the getter/setter exports out of the now-empty generated/includes.jl; keys are immutable, so the metadata-only setters are dropped. Validated single-process: SingleTimeSeries/Deterministic add+read, key lookup (typed + filtered enumeration), accessors, retrieval-by-key, ForecastCache, and remove-by-key. A pre-existing segfault in the concrete DST-by-attributes read (get_time_series(DeterministicSingleTimeSeries, owner, name)) is unrelated to this change (reproduces on the prior branch) and is tracked for a separate fix. BREAKING: removes the *Metadata types and get_time_series_metadata; address time series via TimeSeriesKey / get_time_series_key / get_time_series_keys. Needs PowerSystems.jl coordination. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collapse time-series *Metadata structs into TimeSeriesKey
The Rust backend now answers the catalog/aggregate queries directly, so the glue stops pulling the whole metadata table into Julia and filtering/grouping by hand. Each `TSS.list_metadata` full scan is replaced by a focused binding call: - list-by-owner: `TSS.list_keys` with the catalog filter (owner / name / features / type); `_row_matches` is deleted. - aggregates: `get_resolutions`, `counts_by_type`, `num_distinct_arrays`, `time_series_counts`, `list_owner_ids`, `static_summary` / `forecast_summary`, `check_static_consistency`, and the filtered `get_forecast_parameters`. - batch update diff/rollback (`time_series_manager.jl`) lists keys instead of metadata rows. Type-match semantics are preserved exactly: `_rust_list_metadata` keeps the `_rust_type_matches` rule (a `Deterministic` query also matches a DST), while `list_owner_ids` / `resolutions` / `list_metadata_with_owner` keep strict `<:` matching (`_rust_subtype_codes`). `resolution`/`interval` are matched in Julia with `Period` equality rather than pushed into the catalog query, so an irregular `Month`/`Year` resolution does not spuriously match the stored millisecond resolution (the Rust store keys on milliseconds). Validated: the time-series test suite passes (5855 pass, 0 fail/error, 3 pre-existing irregular-period `@test_broken`). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SingleTimeSeries is now an immutable SingleTimeSeries{T,N} with an
explicit initial_timestamp and a plain data::Array{T,N} (was a
TimeArray); the internal/UUID field is dropped (identity is the array
content hash under the key-centric model). New accessors get_array (raw
Array) and get_time_array (rebuilt TimeArray, N in {1,2}); get_data is a
temporary alias of get_time_array. Regularity is validated at
construction. All TimeArray-backed methods are reimplemented against
(initial_timestamp, resolution, data).
Deterministic/Probabilistic/Scenarios/DeterministicSingleTimeSeries are
now immutable and parameterized on {T,N} (T = element type, N = per-window
array rank; data is SortedDict{DateTime,Array{T,N}}), with internal and
all setters removed. Deterministic keeps a validating inner constructor.
The Rust bridge (serialize_single!/get_single/_rust_get_time_series) uses
get_array and builds structs directly from the decoded array, dropping
the TimeArray hops. Fixes _copy_time_series! to rebuild rather than mutate
the now-immutable name. Removes dead setter exports.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… empty struct gen) - Cost-alias value curves (LinearCurve/QuadraticCurve/Piecewise*) and the CostCurve/FuelCurve wrappers rendered their type names module-qualified on Julia 1.12 (e.g. InfrastructureSystems.QuadraticCurve). Strip the module prefix via simple_type_name/strip_module_name in the compact show methods and add a compact show for ProductionVariableCostCurve that renders a cost-alias type parameter unqualified. - _rust_forecast_parameters converted the stored horizon/interval/resolution to Dates.Millisecond, which throws for irregular Month/Year forecast resolutions now that the store preserves calendar-aware Periods. Pass the Periods through unchanged (ForecastParameters fields are Dates.Period). - test_generated_structs crashed on readdir of a non-existent src/generated directory (the descriptor now defines no auto-generated structs, so the dir is absent). Treat a missing dir as empty and exclude the derived includes.jl manifest from the comparison. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`_rust_get_forecast` previously fetched every window of a forecast via `get_time_series` and then sliced the requested `(start_time, count)` range in Julia with `_forecast_window_range`. Replace that helper with `_forecast_time_range`, which converts the selection into the core's half-open `[start, end)` `time_range` and pushes it into `get_time_series(...; time_range=)`. The store's `resolve_windows` now slices server-side, so the four forecast branches (Probabilistic, Deterministic, DeterministicSingleTimeSeries, Scenarios) iterate the already-sliced result instead of re-deriving window indices, and no longer transfer windows only to discard them. Bounds/alignment/over-request validation stays Julia-side on purpose: the store silently truncates an over-request, so an explicit ArgumentError would otherwise be lost. No Rust/FFI/TimeSeriesStore.jl change — the `time_range` path already ships end to end. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Code-review fixes on the exact-identity branch:
- Keyed reads (get_time_series and the array/values/timestamps accessors)
map a stale key's store NotFoundError to the accessors' documented
ArgumentError, matching what this branch already does on the removal path.
- _infrastore_query_types' empty tuple means "every type" or "no type";
the has_time_series/has_any probes now disambiguate with the same
membership test the bulk removal path uses (shared as
_infrastore_matches_any_stored_type), so a parameterized concrete query
such as SingleTimeSeries{Float64, 1} answers false instead of probing
unfiltered and answering true for the wrong type.
- The name-less has_time_series(owner; kwargs...) form forwards
resolution/interval/feature filters to the store instead of silently
dropping them; the filtered probe accepts name = nothing.
- discard! logs at debug, not error, when the rollback finds no open
transaction — the store already ended it (e.g. a commit that became
durable before commit! threw), so warning about retained partial work
would be false.
- infrastore_remove_time_series! maps InvalidParameterError to the
DST-orphan message only for a SingleTimeSeries key; for any other key
type that diagnosis is impossible and the error propagates as itself.
- infrastore_get_time_series_hash pushes the key's resolution and interval
into the catalog query (canonical store comparisons) instead of
re-implementing the matching client-side; _infrastore_interval_matches
is gone.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
18b363b removed docs/src/dev_guide/associations_database.md (the SQLite associations database it described was replaced by the Rust store) but left the page listed in make.jl, so Documenter fails with "not an existing page" and the docs CI job has been red since. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Documenter's strict cross-reference check failed on six @ref targets: TimeSeriesManager, TimeSeriesContext, SharedSystemReferences, the RelativeUnits module, and add_supplemental_attribute! had no docstring for the referring docstrings to land on, and dev_guide/time_series.md's TimeSeriesKey link needed the module qualifier the page's other links already use. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Components and supplemental attributes each had their own counter on SystemData, so a component and an attribute could share a numeric id. Collapse the two into a single `next_id`, making an id name exactly one object of either kind within a system. The two `assign_id!` methods had identical bodies once the counter merged, so they become one method over the union of the two types. Serialized systems written before this change carry the old `next_component_id`/`next_supplemental_attribute_id` keys. Deserializing one now raises a DataFormatError naming them rather than defaulting the counter to 1, which would silently hand out ids colliding with the restored components. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018EAXp9EmbP5tyuuhkmL5AP
Draw component and supplemental attribute ids from one stream
…act-identity Match the full key identity in key-addressed remove, read, and hash paths
…bles
The time series and supplemental-attribute association catalogs both live in one
InfraStore store, and both are now readable in bulk: one query returns every row,
rather than one query per owner.
For time series this is a correctness requirement before a speed one.
TimeSeriesAssociation requires element_type, element_shape, and address, and none
of the three is reachable from a TimeSeriesKey — InfraStore derives the element
typing from the array it wrote, and re-deriving it here would be a second source
of truth for the one field the schema says the writing package owns. Reading per
owner also meant materializing every series just to reach units, quantity_kind,
and unit_system, three scalars the catalog already stores as columns.
to_openapi gains a method per stored time series type and one for the attachment
row. Document ids are supplied by the caller, matching to_openapi(::GeographicInfo,
::Int): the counter belongs to the document, which is a domain-package concern.
attribute_type comes off the store row, which recorded it at attach time.
begin_association_batch defers association inserts and writes them in one call.
Attaching previously cost two store round trips per row — a has_association probe
so the caller could raise a domain-specific error, then the insert. The probe now
reads the pending buffer as well as the store, so callers keep their error
messages. The batch is write-only while open, does not nest, and writes nothing
if the body throws.
Two fixes on the per-component accessor path, independent of documents:
_type_names expanded the ROOT abstract types into every concrete subtype name on
every call — a subtypes() walk plus a fresh Vector{String} — to build a filter
that matches everything; both roots now answer "no filter", which is the same
query and is the shape get_supplemental_attributes(component) uses.
get_supplemental_attribute probed each type dict through a Union{Nothing,T}
sentinel; now haskey plus index.
test/Project.toml gains its own [sources]. [sources] applies only to the
top-level project and is not inherited by an environment that devs this package,
so without it the test environment cannot resolve the unregistered OpenAPI
packages at all. Aqua's persistent_tasks check is disabled for the same reason
and cannot be re-enabled until PowerOpenAPIModels is registered: it loads the
package in a throwaway project built by Pkg.develop, which hits the identical
resolution failure before it ever looks for a task.
Three related change sets, all green under the full suite (9002 pass, 3 known-broken multi-interval markers): 1. One id counter for components AND supplemental attributes (SystemData.next_entity_id replaces next_component_id + next_supplemental_attribute_id). The store contract is unchanged — InfraStore keeps owner_category and still supports independent id streams from other producers. IS unifies allocation because OpenAPI documents and SiennaGridDB's entities table have a single id space: with overlapping streams an attribute's document id had to be minted fresh and could never resolve against the sidecar catalog (keyed by IS ids), which forced PSY's attribute-owned-time-series export guard and the read-only replay machinery. With one stream an attribute's document id is its IS id exactly like a component's. Full rationale on the field docstring. Both assign_id! methods are now symmetric: keep a pre-set id and bump the counter past it. 2. Serde review fixes (A/B series): begin_association_batch(f, SystemData) wraps begin_supplemental_attributes_update so a mid-batch throw cannot orphan an attached-but-unassociated attribute; _openapi_duration dispatches per period kind and matches the store's ISO spellings including fractional seconds and calendar spans; restore_associations! runs inside InfraStore.transaction; get_non_sequential gained time_range push-down (half-open, far-future sentinel because typemax(DateTime) overflows the FFI millisecond range); component.jl returns T[] not [T]; batch-blindness documented. 3. SupplementalAttributeAssociation document rows now mirror the store row field-for-field: entity_id renamed to component_id and the denormalized component_type label added, both read off the store row (SiennaSchemas change of the same date; generated model packages regenerated).
Remove the pages entry for dev_guide/associations_database.md, deleted in 18b363b but still listed, which failed every build since. Register the docstrings the remaining pages cross-reference: SharedSystemReferences, TimeSeriesManager, RelativeUnits, _association_rows, and the SystemData add_supplemental_attribute! method gain docstrings; TimeSeriesContext's existing docstring was orphaned onto AUTO_FLUSH_THRESHOLD by two const statements and now sits on the struct; one unqualified TimeSeriesKey ref is module-qualified. No new exports. Build finishes clean.
Consolidate the duplicated static-series wrappers onto StaticTimeSeries (labels now carry through every data-sharing constructor; iteration and indexing read the stored data instead of rebuilding a TimeArray); replace isa/<: branches and ternaries with dispatch; drop dead helpers and the sentinel-heavy accessors; restore get_data_store and route the bare manager reaches through it; rename compare_uuids to compare_ids for the integer-id line; trim change-narrating comments, plan citations, and cross-package references; restore the formatter-mangled docs examples; make the on-disk serialization tests run unconditionally instead of silently skipping when the store library is absent.
The store removed reconcile_time_series_associations_openapi: infrastore never modifies the associations table or the data, and a declared-vs-actual geometry mismatch now fails the addition loudly. The wrapper and its policy tests go with it; a rejection test pins the new contract through the public add path (an explicit scenario_count disagreeing with the data errors and writes nothing). Also carries this file pair's share of the review cleanup (row-deserialize helper, kwargs passthrough, store-access routing).
The parity-port skip claimed multi-interval DSTs need interval in the store key; interval has been part of the store identity since, and the whole flow works — coexisting transforms, per-interval has/get, correct window offsets, and an ArgumentError on the ambiguous read. The suite's last Broken entry goes away.
Add bulk catalog reads and OpenAPI converters for both association tables
jd-lara
marked this pull request as ready for review
August 22, 2026 01:39
Bump InfraStore to v0.8.0
jd-lara
self-requested a review
August 22, 2026 16:18
jd-lara
approved these changes
Aug 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.