Optimize and migrate ISD, IBTrACS, CFS reforecast, and OPERA data sources on obstore - #1058
Merged
Conversation
negin513
added a commit
to negin513/earth2studio
that referenced
this pull request
Aug 12, 2026
Revert the ISD, IBTrACS, CFS reforecast, and OPERA obstore migrations to their main state; they now land via NVIDIA#1058 so this PR is NNJA IR only.
6 tasks
Contributor
Greptile SummaryThis PR migrates the ISD, IBTrACS, CFS reforecast, and OPERA remote-read paths from fsspec/s3fs to shared obstore utilities while retaining their URL-derived cache keys.
|
| Filename | Overview |
|---|---|
| earth2studio/data/cfs_reforecast.py | Migrates full-file CFS downloads to the shared obstore cache helper while preserving URI-derived cache keys and missing-file handling. |
| earth2studio/data/ibtracs.py | Migrates region downloads and Last-Modified checks to obstore; legacy metadata may trigger one initial refresh but subsequent cache validation remains stable. |
| earth2studio/data/isd.py | Migrates station-year and station-catalog S3 reads to anonymous obstore stores while preserving cached parquet behavior. |
| earth2studio/data/opera.py | Migrates OPERA archive reads to an HTTP object store while retaining full-URL cache hashes and existing concurrent fetch behavior. |
| CHANGELOG.md | Records the four data-source transport migrations. |
Reviews (1): Last reviewed commit: "Add CHANGELOG entry for ISD/IBTrACS/CFS-..." | Re-trigger Greptile
The NCEI reforecast archive moved from /data/climate-forecast-system/access/reforecast (which now only serves a readme with the new location) to the S3-style endpoint at /oa/prod-cfs-reforecast, with renamed product subdirectories: 6-hourly_9mon_pgbf and cfs_reforecast_6-hourly_9mon_flxf. File naming and date-path layout are unchanged. Verified both pgbf and flxf URIs resolve with HTTP 200 and full fetches decode correctly.
pygrib.select() rescans every message in the file per call (~1.2 s on a 524-message pgbf file), so decoding the full 81-variable lexicon cost 81 sequential scans (~100 s) while the 22 MB download itself takes ~2 s. Replace the per-variable select with one iteration over the file's messages that captures the first message matching each requested (parameterName, typeOfLevel, level) key, preserving select()'s matches[0] semantics. Full-lexicon fetch times (3 reps, cache off): pgbf (81 vars): ~102 s -> ~5 s flxf (13 vars): ~4.7 s -> ~1.6 s Decoded arrays verified bit-identical to the select() path for all 81 pgbf variables.
GFS, HRRR, GEFS, and CFS each carried a private module-level _decode_*_grib helper; the first three were byte-identical and the CFS variant only added a 1-based submessage index for vector wind packings. Replace all four with a shared decode_grib_message(grib_file, message_index=1) in earth2studio.data.utils. pygrib is imported inside the helper so it stays off the import path of non-grib data sources. Offline tests now patch the imported name in each source module.
for more information, see https://pre-commit.ci
This was referenced Aug 12, 2026
negin513
added a commit
that referenced
this pull request
Aug 13, 2026
…at (#1046) * Migrate ISD data source to obstore * Migrate IBTrACS data source to obstore * Migrate CFS reforecast data source to obstore * Migrate OPERA data source to obstore * Update CHANGELOG for obstore data source migrations * feat(data): add NNJAObsIRSat for AIRS, IASI, CrIS from NNJA archive Adds hyperspectral IR sounder support to the NNJA data sources: - earth2studio/data/utils_ir.py: Planck inversion, IASI/CrIS spectral grids, AIRS wavenumber table, ir32/ir48 channel presets (ported from healda/obs/ir_spectral.py) - earth2studio/data/airs_wavenumbers.csv, ir_channel_ranking.csv: bundled data tables copied from healda - earth2studio/data/utils_ncep.py: decode_ir_sounder() parallels decode_microwave(); handles AIRS TMBR (BT direct), IASI SCRA+CHSF (integer radiance with per-band scale), CrIS SRAD (float radiance) - earth2studio/data/nnja_ir.py: NNJAObsIRSat class, follows NNJAObsSat async fetch pattern; ir_channels param selects channel subset - earth2studio/lexicon/nnja_ir.py: NNJAObsIRSatLexicon S3 paths (verified against live bucket): AIRS: airs/airsev/YYYY/MM/bufr/gdas.YYYYMMDD.tHHz.airsev.tm00.bufr_d IASI: iasi/mtiasi/YYYY/MM/bufr/gdas.YYYYMMDD.tHHz.mtiasi.tm00.bufr_d CrIS: cris/cris/YYYY/MM/bufr/gdas.YYYYMMDD.tHHz.cris.tm00.bufr_d Compose with NNJAObsSat via RoutedObsSource (PR #974): RoutedObsSource({ ("atms", "mhs", "amsua", "amsub"): NNJAObsSat(), ("airs", "iasi", "cris"): NNJAObsIRSat(ir_channels="ir32"), }) * style: black/ruff/mypy cleanup for NNJAObsIRSat * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fold NNJAObsIRSat into NNJAObsSat per review feedback Review asked for a single satellite entry point matching the UFS pattern, no HealDA-specific channel presets, and no separate IR class: - NNJAObsSat now serves airs/iasi/cris alongside the microwave sensors: the IR products join _NNJA_SAT_PRODUCTS (identical URI layout) and _decode_file routes IR sensors to decode_ir_sounder with per-sensor platform narrowing. No constructor changes. - Dropped the ir_channels argument, ir_channel_ranking.csv, and the preset helpers; every published channel is returned and selection happens downstream. airs_wavenumbers.csv stays: it populates the schema's wavenumber column for AIRS, whose 281-channel grid is irregular and absent from the BUFR (IASI/CrIS grids are formulaic). - decode_ir_sounder now returns Arrow-typed frames via _rows_to_dataframe so mixed MW+IR requests concat with one dtype contract; per-file fetch timeout raised to 300s for the larger IR aggregates; CSVs added to the sdist include. - Removed nnja_ir.py and its lexicon; airs/iasi/cris live in NNJAObsSatLexicon as sensor::MNEMONIC routes. - Tests: IR task routing/URIs, pre-first_year archive gating, decode routing incl. satellite narrowing, and physically-verified AIRS passthrough + IASI CHSF/Planck + CrIS band-wavenumber decodes, plus utils_ir spectral-grid and Planck unit tests. * Fix IR archive routes and decode defects found in adversarial review Verified against the live noaa-reanalyses-pds bucket and real BUFR decodes: - Route cris to cris/crisf4 (FSR, 2018-present). The previously routed cris/cris archive is the NSR product with different channel numbering, which the FSR conversion grid silently mis-converted for channels >= 714; it also stopped publishing in April 2020. crisf4 verified to carry 431 channels numbered on the 2211-channel FSR grid. - Correct archive coverage gates: airsev is 2007-2020 (not 2002+), mtiasi starts 2008 (not 2007). Products gain an optional last_year so out-of-coverage requests raise archive_unavailable at planning time instead of remote_file_missing after a wasted fetch. - Terminate the IR channel loop at the second replication block: airsev appends AMSU-A/HSB blocks reusing the TMBR descriptor, whose microwave values previously leaked into the output as airs rows. - decode_ir_sounder now raises on failed messages like decode_microwave so the folded source keeps one strict completeness contract. - Add sensor_indices constructor parameter mirroring the JPSS/METOP channel-selection interface (default None = all channels). - Skip planning IR tasks when the satellites filter excludes every platform of a sensor, avoiding a multi-GB fetch that would return zero rows. - Docstring/unit-notation corrections and pinned Planck constants in tests. * Add end-to-end mixed microwave+IR fetch test for NNJAObsSat Exercises the full fetch pipeline for a request spanning both sensor families with mocked downloads/decodes: one Arrow-typed frame, shared schema and dtype contract across the concat. * Read AIRS wavenumbers from the BUFR, dropping airs_wavenumbers.csv The airsev aggregate encodes the channel centre wavenumber per channel as LOGRCW (descriptor 025076, log10 of the wavenumber in 1/m). Decoding it removes the need for the bundled CSV table entirely; verified on a real 2019 file that all 281 channels resolve with values matching the old table to <0.01 1/cm. The same real-file check exposed that the AMSU-A/HSB blocks trailing the AIRS channels are not always preceded by another replication descriptor, so the second-31002 terminator was insufficient — with LOGRCW decoded, leaked microwave channels would have carried plausible- looking wavenumbers. The declared replication count is now honored as the channel-block budget, which closes the leak for the real layout (exactly 281 channels decode, no extras). IASI/CrIS decode unchanged (verified byte-identical on a real crisf4 file). Also removes the previous dead SCCF wavenumber path and fixes the SCRA unit notation in the iasi_radiance_mw docstring. * Address IR sounder review: AIRS route, CrIS FORN/quality, uint32 quality - Route AIRS to airs/nasa/aqua (2002-2023) instead of airs/airsev (2007-2020); the NASA route is structurally identical and spans five more early years plus the instrument's final two. Adds a per-product naming convention since the NASA files are named airs_disc_final.YYYYMMDD.tHHz.bufr rather than gdas.*.tm00.bufr_d. Verified against the live bucket. - Emit CrIS FORN (cross-track, 1-30) as scan_position; FOVN is the 3x3 within-FoR detector index (1-9) and is not the cross-track position. - Decode CrIS per-band NFQF/NCQF quality flags: resolve each channel's band and pack nfqf | ncqf << 19 into the quality column. NFQF is the flag that fires on real sensor failures (e.g. the NOAA-21 neon event) whose damaged radiances are otherwise finite and plausible. - Widen quality to uint32: AIRS ACQF is a 24-bit flag table, and a value above 65535 raised ArrowInvalid outside the per-message try, failing the whole request. * Deduplicate Planck inversion; pin CrIS grid against jpss_cris - Drop C1/C2/brightness_temperature from utils_ir.py in favor of the shared radiance_to_bt / PLANCK_C1 / PLANCK_C2 in utils.py, already used by the MetOp and JPSS granule sources for the same instruments. - Switch the shared inversion from np.log1p to np.log(1 + x): the argument never falls below ~16 over Earth scenes, so log1p buys no accuracy and is measurably slower. - Correct the radiance_to_bt docstring, which claimed CODATA 2018 but printed the 2014 values. - Add a cross-consistency test asserting wavenumber_cm_inverse("cris") matches jpss_cris._CRIS_WAVENUMBER_APOD element for element, and that the IASI channel count matches metop_iasi, so the grids cannot drift apart silently. * Harden IR sounder decode: validation, guard markers, docs - Validate sensor_indices at construction (non-empty, unique, integer, within the instrument grid) so an empty or out-of-range selection raises instead of silently returning an empty frame after downloading a multi-GB aggregate; matches the JPSS/METOP sources' behavior. - Recognize DRF8BIT (0-31-001) as the crisf4 guard-spectrum block boundary and scope the second-DRF16BIT break to AIRS, so the guard protection is explicit and a nested replication block in a future IASI/CrIS product cannot silently truncate footprints. The comment now states the channel budget is the primary terminator with markers as backstops. - Drop footprints missing their cross-track position (FORN for CrIS, FOVN otherwise) instead of emitting None into the non-nullable scan_position column, matching the microwave path. - Resolve IASI/CrIS channel wavenumbers once per footprint instead of once per channel, removing a one-element numpy allocation per channel from the hot loop; hoist the conversion imports to module scope. - Log a debug count of IASI channels skipped for missing CHSF bands so a narrower-than-expected band table is distinguishable from no data. - Documentation: elev is CrIS-only (SELV is platform altitude, not a substitute); scan_angle is always NaN for IR; per-sensor scan_position semantics; real per-cycle row magnitudes in the memory warning; CHSF is a channel-group exponent, not per-band. * Split obstore migrations out to their own PR Revert the ISD, IBTrACS, CFS reforecast, and OPERA obstore migrations to their main state; they now land via #1058 so this PR is NNJA IR only. * Post-review polish: quality metadata, docstring placement, worker guard - Correct the quality column metadata: the microwave path populates it from the per-channel 0-33-081 flags, so "MW: null" was wrong; only IASI leaves it null. - Move the utils_ir module docstring into a comment block below the license header per e2s-002 (only license headers above imports). - Raise on an uninitialized BUFR decoder worker in the IR batch fn, matching the microwave path, instead of miscounting it as per-message decode failures. * Update NNJAObsSat lexicon test for AIRS, IASI, CrIS entries --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
6 tasks
Member
Author
|
/ok to test 8e5c768 |
negin513
changed the title
negin513
changed the title
Member
Author
|
/ok to test dc5c5a4 |
Member
Author
|
/ok to test ff59548 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Earth2Studio Pull Request
Description
Closes NVIDIA/physicsnemo-roadmap#2883
closes NVIDIA/physicsnemo-roadmap#2886
closes NVIDIA/physicsnemo-roadmap#2887
closes NVIDIA/physicsnemo-roadmap#2888
Migrates four data sources from fsspec/s3fs/HTTP to obstore, continuing the migration pattern established by the GOES (#1024), GOES GLM, and Himawari AHI (#1043) PRs:
Each migration is a single-file commit. Cache-key formats are unchanged, so existing local caches remain valid.
Three additional changes surfaced by benchmarking this PR:
/data/climate-forecast-system/access/reforecastpath (it now serves only a readme), breakingCFS_Reforecast_FX/CFS_Reforecast_FX_Fluxonmainas well. Both classes now point at the relocated archive (https://www.ncei.noaa.gov/oa/prod-cfs-reforecast) with its renamed product subdirs; file naming and date-path layout are unchanged.pygrib.select()once per variable, and each call rescans every message in the file (~1.2 s on a 524-message pgbf file), costing ~100 s for a full-lexicon request while the 22 MB download takes ~2 s. Variables are now resolved in one pass over the file's messages, preservingselect()'s first-match semantics. Decoded arrays verified bit-identical to the old path for all 81 pgbf variables._decode_*_gribhelper; all four are consolidated intodecode_grib_messageinearth2studio.data.utils(net −77 lines). Pure refactor: identical decode logic and thread dispatch, no performance or behavior change for those sources.Performance
Each source fetched its full lexicon with
cache=False, 3 reps, same host and time window; medians shown. CFS rows used the relocated NCEI endpoint on both sides for an apples-to-apples comparison.The CSV/netCDF/HDF5 sources gain from obstore's lower per-request overhead. The GRIB2 sources download one whole file per (time, lead) — the reforecast bucket has no
.idxcompanions — so their gain comes from the single-pass decode, not the transfer layer.The
.idxbyte-range sources (GFS, HRRR, GEFS, CFS operational) are untouched by this PR beyond the helper consolidation. Batching their per-variable range requests throughobstore.get_ranges(which coalesces ranges under 1 MiB apart) was prototyped and benchmarked, and rejected: full-lexicon CFS_FX (81 vars) was a wash (~3.6 s both ways) and full-lexicon GFS (246 vars) regressed ~10–15% (12.1–12.8 s → 13.3–26.5 s), becauseget_rangescaps at 10 parallel fetches per file while the existing per-range path fans out across up to 64 async workers.Checklist
test_isd.py,test_ibtracs.py,test_cfs.py,test_cfs_reforecast.py,test_opera.pypass).Dependencies
No new dependencies; obstore and obspec are already core dependencies.