Add hyperspectral IR sounder variables (AIRS, IASI, CrIS) to NNJAObsSat - #1046
Conversation
Adds hyperspectral IR sounder support to the NNJA data sources: - earth2studio/data/utils_ir.py: Planck inversion, IASI/CrIS spectral grids, AIRS wavenumber table, ir32/ir48 channel presets (ported from healda/obs/ir_spectral.py) - earth2studio/data/airs_wavenumbers.csv, ir_channel_ranking.csv: bundled data tables copied from healda - earth2studio/data/utils_ncep.py: decode_ir_sounder() parallels decode_microwave(); handles AIRS TMBR (BT direct), IASI SCRA+CHSF (integer radiance with per-band scale), CrIS SRAD (float radiance) - earth2studio/data/nnja_ir.py: NNJAObsIRSat class, follows NNJAObsSat async fetch pattern; ir_channels param selects channel subset - earth2studio/lexicon/nnja_ir.py: NNJAObsIRSatLexicon S3 paths (verified against live bucket): AIRS: airs/airsev/YYYY/MM/bufr/gdas.YYYYMMDD.tHHz.airsev.tm00.bufr_d IASI: iasi/mtiasi/YYYY/MM/bufr/gdas.YYYYMMDD.tHHz.mtiasi.tm00.bufr_d CrIS: cris/cris/YYYY/MM/bufr/gdas.YYYYMMDD.tHHz.cris.tm00.bufr_d Compose with NNJAObsSat via RoutedObsSource (PR NVIDIA#974): RoutedObsSource({ ("atms", "mhs", "amsua", "amsub"): NNJAObsSat(), ("airs", "iasi", "cris"): NNJAObsIRSat(ir_channels="ir32"), })
Greptile SummaryAdds
|
| Filename | Overview |
|---|---|
| earth2studio/data/nnja_ir.py | Implements the NNJA hyperspectral IR dataframe source, request validation, S3 fetching, caching, task construction, and decode orchestration. |
| earth2studio/data/utils_ncep.py | Adds AIRS, IASI, and CrIS BUFR decoding and brightness-temperature conversion integration; no concrete changed-code failure was established. |
| earth2studio/data/utils_ir.py | Defines spectral grids, channel presets, radiance-unit conversions, and Planck inversion for the three supported sounders. |
| earth2studio/lexicon/nnja_ir.py | Adds sensor-name mappings for the new dataframe source. |
| earth2studio/data/cfs_reforecast.py | Migrates full-file HTTP fetching to obstore while preserving historical cache keys. |
| earth2studio/data/ibtracs.py | Migrates HTTP downloads and Last-Modified cache validation to obstore. |
| earth2studio/data/isd.py | Migrates station CSV and station-history S3 reads from s3fs to obstore. |
| earth2studio/data/opera.py | Migrates OPERA HTTP object reads to obstore while retaining URL-derived cache keys. |
Reviews (1): Last reviewed commit: "style: black/ruff/mypy cleanup for NNJAO..." | Re-trigger Greptile
for more information, see https://pre-commit.ci
|
@negin513 Some initial high level comments:
|
# Conflicts: # CHANGELOG.md
Review asked for a single satellite entry point matching the UFS pattern, no HealDA-specific channel presets, and no separate IR class: - NNJAObsSat now serves airs/iasi/cris alongside the microwave sensors: the IR products join _NNJA_SAT_PRODUCTS (identical URI layout) and _decode_file routes IR sensors to decode_ir_sounder with per-sensor platform narrowing. No constructor changes. - Dropped the ir_channels argument, ir_channel_ranking.csv, and the preset helpers; every published channel is returned and selection happens downstream. airs_wavenumbers.csv stays: it populates the schema's wavenumber column for AIRS, whose 281-channel grid is irregular and absent from the BUFR (IASI/CrIS grids are formulaic). - decode_ir_sounder now returns Arrow-typed frames via _rows_to_dataframe so mixed MW+IR requests concat with one dtype contract; per-file fetch timeout raised to 300s for the larger IR aggregates; CSVs added to the sdist include. - Removed nnja_ir.py and its lexicon; airs/iasi/cris live in NNJAObsSatLexicon as sensor::MNEMONIC routes. - Tests: IR task routing/URIs, pre-first_year archive gating, decode routing incl. satellite narrowing, and physically-verified AIRS passthrough + IASI CHSF/Planck + CrIS band-wavenumber decodes, plus utils_ir spectral-grid and Planck unit tests.
Verified against the live noaa-reanalyses-pds bucket and real BUFR decodes: - Route cris to cris/crisf4 (FSR, 2018-present). The previously routed cris/cris archive is the NSR product with different channel numbering, which the FSR conversion grid silently mis-converted for channels >= 714; it also stopped publishing in April 2020. crisf4 verified to carry 431 channels numbered on the 2211-channel FSR grid. - Correct archive coverage gates: airsev is 2007-2020 (not 2002+), mtiasi starts 2008 (not 2007). Products gain an optional last_year so out-of-coverage requests raise archive_unavailable at planning time instead of remote_file_missing after a wasted fetch. - Terminate the IR channel loop at the second replication block: airsev appends AMSU-A/HSB blocks reusing the TMBR descriptor, whose microwave values previously leaked into the output as airs rows. - decode_ir_sounder now raises on failed messages like decode_microwave so the folded source keeps one strict completeness contract. - Add sensor_indices constructor parameter mirroring the JPSS/METOP channel-selection interface (default None = all channels). - Skip planning IR tasks when the satellites filter excludes every platform of a sensor, avoiding a multi-GB fetch that would return zero rows. - Docstring/unit-notation corrections and pinned Planck constants in tests.
Exercises the full fetch pipeline for a request spanning both sensor families with mocked downloads/decodes: one Arrow-typed frame, shared schema and dtype contract across the concat.
|
Thanks @aayushg55, good points — went ahead and restructured:
|
The airsev aggregate encodes the channel centre wavenumber per channel as LOGRCW (descriptor 025076, log10 of the wavenumber in 1/m). Decoding it removes the need for the bundled CSV table entirely; verified on a real 2019 file that all 281 channels resolve with values matching the old table to <0.01 1/cm. The same real-file check exposed that the AMSU-A/HSB blocks trailing the AIRS channels are not always preceded by another replication descriptor, so the second-31002 terminator was insufficient — with LOGRCW decoded, leaked microwave channels would have carried plausible- looking wavenumbers. The declared replication count is now honored as the channel-block budget, which closes the leak for the real layout (exactly 281 channels decode, no extras). IASI/CrIS decode unchanged (verified byte-identical on a real crisf4 file). Also removes the previous dead SCCF wavenumber path and fixes the SCRA unit notation in the iasi_radiance_mw docstring.
|
Update on the airs csv: turns out it's not needed at all. The airsev BUFR encodes the centre wavenumber per channel (LOGRCW, log10 of the wavenumber in m⁻¹), so I now read it straight from the file and deleted the csv. Checked against a real 2019 file, all 281 channels match the old table to <0.01 cm⁻¹. Doing that also surfaced a decode bug. The AMSU-A/HSB blocks that follow the AIRS channels aren't always separated by another replication marker, so a few microwave channels were leaking through as AIRS rows. The decoder now uses the declared replication count as the block boundary. |
|
@negin513 Thanks for the restructure. I had my agent review the NNJA IR based on our current work/findings in converting the same bufr to parquet. It produced a rather lengthy document which I think has some highly relevant information which would be hard to distill otherwise, so attaching in its full here as documentation/reference. The summary and the suggested changes are both at the top, everything after that is the additional evidence. The four correctness items, repeated here:
The rest is mainly nice to have code deduplication and some comments worth correcting. There are also some points on performance/memory, but this is not really specific to this implementation but rather generally applicable for all these raw high-resolution MW/IR sounders data sources, and particularly for IR which has hundreds of channels. These would probably be best addressed in future PRs. TLDR, as is, 48hr of satellite data might consume a lot of memory. |
- Route AIRS to airs/nasa/aqua (2002-2023) instead of airs/airsev (2007-2020); the NASA route is structurally identical and spans five more early years plus the instrument's final two. Adds a per-product naming convention since the NASA files are named airs_disc_final.YYYYMMDD.tHHz.bufr rather than gdas.*.tm00.bufr_d. Verified against the live bucket. - Emit CrIS FORN (cross-track, 1-30) as scan_position; FOVN is the 3x3 within-FoR detector index (1-9) and is not the cross-track position. - Decode CrIS per-band NFQF/NCQF quality flags: resolve each channel's band and pack nfqf | ncqf << 19 into the quality column. NFQF is the flag that fires on real sensor failures (e.g. the NOAA-21 neon event) whose damaged radiances are otherwise finite and plausible. - Widen quality to uint32: AIRS ACQF is a 24-bit flag table, and a value above 65535 raised ArrowInvalid outside the per-message try, failing the whole request.
- Drop C1/C2/brightness_temperature from utils_ir.py in favor of the
shared radiance_to_bt / PLANCK_C1 / PLANCK_C2 in utils.py, already
used by the MetOp and JPSS granule sources for the same instruments.
- Switch the shared inversion from np.log1p to np.log(1 + x): the
argument never falls below ~16 over Earth scenes, so log1p buys no
accuracy and is measurably slower.
- Correct the radiance_to_bt docstring, which claimed CODATA 2018 but
printed the 2014 values.
- Add a cross-consistency test asserting wavenumber_cm_inverse("cris")
matches jpss_cris._CRIS_WAVENUMBER_APOD element for element, and that
the IASI channel count matches metop_iasi, so the grids cannot drift
apart silently.
- Validate sensor_indices at construction (non-empty, unique, integer, within the instrument grid) so an empty or out-of-range selection raises instead of silently returning an empty frame after downloading a multi-GB aggregate; matches the JPSS/METOP sources' behavior. - Recognize DRF8BIT (0-31-001) as the crisf4 guard-spectrum block boundary and scope the second-DRF16BIT break to AIRS, so the guard protection is explicit and a nested replication block in a future IASI/CrIS product cannot silently truncate footprints. The comment now states the channel budget is the primary terminator with markers as backstops. - Drop footprints missing their cross-track position (FORN for CrIS, FOVN otherwise) instead of emitting None into the non-nullable scan_position column, matching the microwave path. - Resolve IASI/CrIS channel wavenumbers once per footprint instead of once per channel, removing a one-element numpy allocation per channel from the hot loop; hoist the conversion imports to module scope. - Log a debug count of IASI channels skipped for missing CHSF bands so a narrower-than-expected band table is distinguishable from no data. - Documentation: elev is CrIS-only (SELV is platform altitude, not a substitute); scan_angle is always NaN for IR; per-sensor scan_position semantics; real per-cycle row magnitudes in the memory warning; CHSF is a channel-group exponent, not per-band.
Revert the ISD, IBTrACS, CFS reforecast, and OPERA obstore migrations to their main state; they now land via NVIDIA#1058 so this PR is NNJA IR only.
- Correct the quality column metadata: the microwave path populates it from the per-channel 0-33-081 flags, so "MW: null" was wrong; only IASI leaves it null. - Move the utils_ir module docstring into a comment block below the license header per e2s-002 (only license headers above imports). - Raise on an uninitialized BUFR decoder worker in the IR batch fn, matching the microwave path, instead of miscounting it as per-message decode failures.
|
/ok to test f4f0679 |
|
Thanks @aayushg55 . All issues are addressed and started a perf PR on #1059 for vectorized conversion. |
|
Will bring this in once all tests pass. |
|
/ok to test cbfb1f6 |
Description
Adds the hyperspectral IR sounders —
airs,iasi,cris— as variables on the existingNNJAObsSatsource, reading the NCEP aggregate BUFR products from the NNJA archive ats3://noaa-reanalyses-pds. One source now covers all NNJA satellite data (microwave + IR) with a single entry point, matching how UFS sat data is structured; a mixed request like["atms", "cris"]returns a single frame.All three sensors are returned in brightness temperature (K):
airs/nasa/aquaroute): stored asTMBR— no conversion needed. Per-channel wavenumbers read from theLOGRCWfield in the BUFR.SCRA) with per-channel-group scale exponents (CHSF); Planck inversion via the sharedradiance_to_bt.crisf4FSR): float radiance (SRAD); Planck inversion. Per-bandNFQF/NCQFquality flags packed into thequalitycolumn; cross-track position fromFORN.Channel selection follows the JPSS/METOP
sensor_indicesinterface (validated at construction: non-empty, unique, on the instrument grid); default returns all archived channels.New files
data/utils_ir.pylexicon/nnja.pyairs/iasi/crisvariable ids (extends existing lexicon)test/data/test_utils_ir.pyModified files
data/utils_ncep.py— addsdecode_ir_sounder()alongsidedecode_microwave(), using the sameProcessPoolExecutor/batch pattern; widensqualitytouint32(AIRSACQFis 24-bit, CrIS packed band flags are 28)data/nnja.py— IR product routing with per-product naming conventions, platform maps,sensor_indiceswith validationdata/utils.py—radiance_to_bt:log1p→log(1+x)(argument ≥16 over Earth scenes; measurably faster, identical accuracy) and CODATA docstring correctionS3 paths (verified against live bucket)
Review history
All four correctness items from the review are addressed:
airs/nasa/aqua(2002–2023) with theairs_disc_finalfilename conventionscan_positionisFORN(1–30) for CrIS,FOVNfor all other sensorsNFQF/NCQFpacked asnfqf | ncqf<<19per channel row; IASI comment reworded to name the footprint-level QGFQ-family scalarsqualitywidened touint32Plus the non-blocking dedup and hardening items: Planck inversion consolidated onto the shared
radiance_to_bt; CrIS spectral grid pinned againstjpss_cris(and IASI channel count againstmetop_iasi) by test;sensor_indicesvalidation;0-31-001guard-spectrum marker handled explicitly with the second-31002break scoped to AIRS; footprints missing their cross-track position dropped rather than failing the file; wavenumbers resolved once per footprint; documentation forelev(CrIS-only;SELVis platform altitude),scan_angle(always NaN for IR), per-sensorscan_positionsemantics, and realistic per-cycle row magnitudes in the memory warning.Note: the ISD/IBTrACS/CFS-reforecast/OPERA obstore migrations previously stacked on this branch were split out to #1058; this PR is now NNJA IR only.
Follow-on PRs
iasi_radiance/cris_radiancevariables alongside BT (§2.2)airs_gdasvariant exposing theairsevroute (collocated AMSU-A + TOCC; §2.3)Checklist
noaa-reanalyses-pdsbucketjpss_cris/metop_iasi)NNJAObsSatentry indatasources_dataframe.rstrenders the updated docstringruffcleanmypy— only pre-existingpandasstubs warning (identical tonnja.py,ufs.py, etc.)