Skip to content

Add hyperspectral IR sounder variables (AIRS, IASI, CrIS) to NNJAObsSat - #1046

Merged
negin513 merged 20 commits into
NVIDIA:mainfrom
negin513:obstore-independent-four
Aug 13, 2026
Merged

Add hyperspectral IR sounder variables (AIRS, IASI, CrIS) to NNJAObsSat#1046
negin513 merged 20 commits into
NVIDIA:mainfrom
negin513:obstore-independent-four

Conversation

@negin513

@negin513 negin513 commented Aug 6, 2026

Copy link
Copy Markdown
Member

Description

Adds the hyperspectral IR sounders — airs, iasi, cris — as variables on the existing NNJAObsSat source, reading the NCEP aggregate BUFR products from the NNJA archive at s3://noaa-reanalyses-pds. One source now covers all NNJA satellite data (microwave + IR) with a single entry point, matching how UFS sat data is structured; a mixed request like ["atms", "cris"] returns a single frame.

All three sensors are returned in brightness temperature (K):

  • AIRS (Aqua, 2002–2023, airs/nasa/aqua route): stored as TMBR — no conversion needed. Per-channel wavenumbers read from the LOGRCW field in the BUFR.
  • IASI (MetOp-A/B/C, 2008–present): integer-coded scaled radiance (SCRA) with per-channel-group scale exponents (CHSF); Planck inversion via the shared radiance_to_bt.
  • CrIS (NPP/NOAA-20/NOAA-21, 2018–present, crisf4 FSR): float radiance (SRAD); Planck inversion. Per-band NFQF/NCQF quality flags packed into the quality column; cross-track position from FORN.

Channel selection follows the JPSS/METOP sensor_indices interface (validated at construction: non-empty, unique, on the instrument grid); default returns all archived channels.

New files

File Purpose
data/utils_ir.py IASI/CrIS spectral grids, radiance unit conversions
lexicon/nnja.py airs/iasi/cris variable ids (extends existing lexicon)
test/data/test_utils_ir.py Grid, conversion, and cross-consistency tests

Modified files

  • data/utils_ncep.py — adds decode_ir_sounder() alongside decode_microwave(), using the same ProcessPoolExecutor/batch pattern; widens quality to uint32 (AIRS ACQF is 24-bit, CrIS packed band flags are 28)
  • data/nnja.py — IR product routing with per-product naming conventions, platform maps, sensor_indices with validation
  • data/utils.pyradiance_to_bt: log1plog(1+x) (argument ≥16 over Earth scenes; measurably faster, identical accuracy) and CODATA docstring correction

S3 paths (verified against live bucket)

AIRS: airs/nasa/aqua/YYYY/MM/bufr/airs_disc_final.YYYYMMDD.tHHz.bufr
IASI: iasi/mtiasi/YYYY/MM/bufr/gdas.YYYYMMDD.tHHz.mtiasi.tm00.bufr_d
CrIS: cris/crisf4/YYYY/MM/bufr/gdas.YYYYMMDD.tHHz.crisf4.tm00.bufr_d

Review history

All four correctness items from the review are addressed:

  1. AIRS routed to airs/nasa/aqua (2002–2023) with the airs_disc_final filename convention
  2. CrIS cross-track position emitted: scan_position is FORN (1–30) for CrIS, FOVN for all other sensors
  3. CrIS quality decoded: per-band NFQF/NCQF packed as nfqf | ncqf<<19 per channel row; IASI comment reworded to name the footprint-level QGFQ-family scalars
  4. quality widened to uint32

Plus the non-blocking dedup and hardening items: Planck inversion consolidated onto the shared radiance_to_bt; CrIS spectral grid pinned against jpss_cris (and IASI channel count against metop_iasi) by test; sensor_indices validation; 0-31-001 guard-spectrum marker handled explicitly with the second-31002 break scoped to AIRS; footprints missing their cross-track position dropped rather than failing the file; wavenumbers resolved once per footprint; documentation for elev (CrIS-only; SELV is platform altitude), scan_angle (always NaN for IR), per-sensor scan_position semantics, and realistic per-cycle row magnitudes in the memory warning.

Note: the ISD/IBTrACS/CFS-reforecast/OPERA obstore migrations previously stacked on this branch were split out to #1058; this PR is now NNJA IR only.

Follow-on PRs

  • perf(data): vectorize NNJA IR decode and accumulate Arrow tables per batch #1059 (draft, stacked) — performance/memory: vectorized per-footprint conversion and Arrow-per-batch accumulation, per §5.1–5.2 of the review analysis
  • Emit ≤0-radiance rows with null observation instead of dropping (provenance; §4.4)
  • iasi_radiance/cris_radiance variables alongside BT (§2.2)
  • Optional airs_gdas variant exposing the airsev route (collocated AMSU-A + TOCC; §2.3)
  • Extract a shared MW/IR batch-decode harness (the two paths are deliberately parallel today)

Checklist

  • S3 paths verified against live noaa-reanalyses-pds bucket
  • Output schema shared with the microwave sensors (mixed MW+IR requests concat cleanly)
  • Unit tests — 67 passing (mocked BUFR decode, grid cross-consistency vs jpss_cris/metop_iasi)
  • Docs — NNJAObsSat entry in datasources_dataframe.rst renders the updated docstring
  • ruff clean
  • mypy — only pre-existing pandas stubs warning (identical to nnja.py, ufs.py, etc.)

Adds hyperspectral IR sounder support to the NNJA data sources:

- earth2studio/data/utils_ir.py: Planck inversion, IASI/CrIS spectral
  grids, AIRS wavenumber table, ir32/ir48 channel presets (ported from
  healda/obs/ir_spectral.py)
- earth2studio/data/airs_wavenumbers.csv, ir_channel_ranking.csv: bundled
  data tables copied from healda
- earth2studio/data/utils_ncep.py: decode_ir_sounder() parallels
  decode_microwave(); handles AIRS TMBR (BT direct), IASI SCRA+CHSF
  (integer radiance with per-band scale), CrIS SRAD (float radiance)
- earth2studio/data/nnja_ir.py: NNJAObsIRSat class, follows NNJAObsSat
  async fetch pattern; ir_channels param selects channel subset
- earth2studio/lexicon/nnja_ir.py: NNJAObsIRSatLexicon

S3 paths (verified against live bucket):
  AIRS:  airs/airsev/YYYY/MM/bufr/gdas.YYYYMMDD.tHHz.airsev.tm00.bufr_d
  IASI:  iasi/mtiasi/YYYY/MM/bufr/gdas.YYYYMMDD.tHHz.mtiasi.tm00.bufr_d
  CrIS:  cris/cris/YYYY/MM/bufr/gdas.YYYYMMDD.tHHz.cris.tm00.bufr_d

Compose with NNJAObsSat via RoutedObsSource (PR NVIDIA#974):
  RoutedObsSource({
      ("atms", "mhs", "amsua", "amsub"): NNJAObsSat(),
      ("airs", "iasi", "cris"):          NNJAObsIRSat(ir_channels="ir32"),
  })
@copy-pr-bot

copy-pr-bot Bot commented Aug 6, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@negin513 negin513 changed the title feat(data): add NNJAObsIRSat for AIRS, IASI, CrIS from NNJA archive wip(data): add NNJAObsIRSat Aug 6, 2026
@greptile-apps

greptile-apps Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Adds NNJAObsIRSat for AIRS, IASI, and CrIS observations, including channel presets, spectral metadata, radiance-to-brightness-temperature conversion, BUFR decoding, and public exports.

  • Adds ranked IR channel and AIRS wavenumber tables.
  • Extends shared NCEP decoding for hyperspectral sounders.
  • Migrates CFS reforecast, IBTrACS, ISD, and OPERA remote reads to obstore.

Confidence Score: 5/5

The PR appears safe to merge; no concrete blocking or independently actionable non-blocking defect was established.

The new IR source, spectral conversion utilities, BUFR integration, exports, and obstore migrations preserve the relevant source, schema, cache, and remote-read contracts on the reachable paths reviewed.

Important Files Changed

Filename Overview
earth2studio/data/nnja_ir.py Implements the NNJA hyperspectral IR dataframe source, request validation, S3 fetching, caching, task construction, and decode orchestration.
earth2studio/data/utils_ncep.py Adds AIRS, IASI, and CrIS BUFR decoding and brightness-temperature conversion integration; no concrete changed-code failure was established.
earth2studio/data/utils_ir.py Defines spectral grids, channel presets, radiance-unit conversions, and Planck inversion for the three supported sounders.
earth2studio/lexicon/nnja_ir.py Adds sensor-name mappings for the new dataframe source.
earth2studio/data/cfs_reforecast.py Migrates full-file HTTP fetching to obstore while preserving historical cache keys.
earth2studio/data/ibtracs.py Migrates HTTP downloads and Last-Modified cache validation to obstore.
earth2studio/data/isd.py Migrates station CSV and station-history S3 reads from s3fs to obstore.
earth2studio/data/opera.py Migrates OPERA HTTP object reads to obstore while retaining URL-derived cache keys.

Reviews (1): Last reviewed commit: "style: black/ruff/mypy cleanup for NNJAO..." | Re-trigger Greptile

@negin513
negin513 requested a review from aayushg55 August 6, 2026 15:27
@aayushg55

Copy link
Copy Markdown
Collaborator

@negin513 Some initial high level comments:

  • I don't think we need a separate NNJAObsIRSat. It would be nice to match the UFS sat data in how the single "source" contains all the satellite data (IR + MW) and there is a single entry point. We can just have a separate decode path for MW and IR. But open to push back here
  • For channel selection, we shouldn't add the csv we created (the channel ranking file). Our HealDA specific things should be largely kept orthogonal to the E2S datasource. Channel selection should happen through the existing interface /api that exists for UFS, JPSS CRIS, and others. (i.e. no ir_channels argument). Default should return all channels
  • not sure if we need airs_wavenumbers.csv as of yet.

Review asked for a single satellite entry point matching the UFS
pattern, no HealDA-specific channel presets, and no separate IR class:

- NNJAObsSat now serves airs/iasi/cris alongside the microwave sensors:
  the IR products join _NNJA_SAT_PRODUCTS (identical URI layout) and
  _decode_file routes IR sensors to decode_ir_sounder with per-sensor
  platform narrowing. No constructor changes.
- Dropped the ir_channels argument, ir_channel_ranking.csv, and the
  preset helpers; every published channel is returned and selection
  happens downstream. airs_wavenumbers.csv stays: it populates the
  schema's wavenumber column for AIRS, whose 281-channel grid is
  irregular and absent from the BUFR (IASI/CrIS grids are formulaic).
- decode_ir_sounder now returns Arrow-typed frames via
  _rows_to_dataframe so mixed MW+IR requests concat with one dtype
  contract; per-file fetch timeout raised to 300s for the larger IR
  aggregates; CSVs added to the sdist include.
- Removed nnja_ir.py and its lexicon; airs/iasi/cris live in
  NNJAObsSatLexicon as sensor::MNEMONIC routes.
- Tests: IR task routing/URIs, pre-first_year archive gating, decode
  routing incl. satellite narrowing, and physically-verified AIRS
  passthrough + IASI CHSF/Planck + CrIS band-wavenumber decodes, plus
  utils_ir spectral-grid and Planck unit tests.
Verified against the live noaa-reanalyses-pds bucket and real BUFR
decodes:

- Route cris to cris/crisf4 (FSR, 2018-present). The previously routed
  cris/cris archive is the NSR product with different channel numbering,
  which the FSR conversion grid silently mis-converted for channels
  >= 714; it also stopped publishing in April 2020. crisf4 verified to
  carry 431 channels numbered on the 2211-channel FSR grid.
- Correct archive coverage gates: airsev is 2007-2020 (not 2002+),
  mtiasi starts 2008 (not 2007). Products gain an optional last_year so
  out-of-coverage requests raise archive_unavailable at planning time
  instead of remote_file_missing after a wasted fetch.
- Terminate the IR channel loop at the second replication block:
  airsev appends AMSU-A/HSB blocks reusing the TMBR descriptor, whose
  microwave values previously leaked into the output as airs rows.
- decode_ir_sounder now raises on failed messages like decode_microwave
  so the folded source keeps one strict completeness contract.
- Add sensor_indices constructor parameter mirroring the JPSS/METOP
  channel-selection interface (default None = all channels).
- Skip planning IR tasks when the satellites filter excludes every
  platform of a sensor, avoiding a multi-GB fetch that would return
  zero rows.
- Docstring/unit-notation corrections and pinned Planck constants in
  tests.
Exercises the full fetch pipeline for a request spanning both sensor
families with mocked downloads/decodes: one Arrow-typed frame, shared
schema and dtype contract across the concat.
@negin513

negin513 commented Aug 11, 2026

Copy link
Copy Markdown
Member Author

Thanks @aayushg55, good points — went ahead and restructured:

  • Dropped NNJAObsIRSat entirely. The IR sounders are now just variables on NNJAObsSat (airs, iasi, cris) and _decode_file routes them to the IR decode path internally. So one source for all the sat data like UFS, and a mixed request like ["atms", "cris"] comes back as a single frame.
  • Removed ir_channels and the ranking csv. Default is all channels now. I added a sensor_indices arg for narrowing since that's the interface JPSS_CRIS and the METOP sources already use, but you only get subsetting if you ask for it.
  • On airs_wavenumbers.csv, what would be the best way to handle it? Concidering missing wavwenumber data?

@negin513
negin513 requested a review from NickGeneva August 11, 2026 05:36
The airsev aggregate encodes the channel centre wavenumber per channel
as LOGRCW (descriptor 025076, log10 of the wavenumber in 1/m). Decoding
it removes the need for the bundled CSV table entirely; verified on a
real 2019 file that all 281 channels resolve with values matching the
old table to <0.01 1/cm.

The same real-file check exposed that the AMSU-A/HSB blocks trailing
the AIRS channels are not always preceded by another replication
descriptor, so the second-31002 terminator was insufficient — with
LOGRCW decoded, leaked microwave channels would have carried plausible-
looking wavenumbers. The declared replication count is now honored as
the channel-block budget, which closes the leak for the real layout
(exactly 281 channels decode, no extras). IASI/CrIS decode unchanged
(verified byte-identical on a real crisf4 file).

Also removes the previous dead SCCF wavenumber path and fixes the SCRA
unit notation in the iasi_radiance_mw docstring.
@negin513

Copy link
Copy Markdown
Member Author

Update on the airs csv: turns out it's not needed at all. The airsev BUFR encodes the centre wavenumber per channel (LOGRCW, log10 of the wavenumber in m⁻¹), so I now read it straight from the file and deleted the csv. Checked against a real 2019 file, all 281 channels match the old table to <0.01 cm⁻¹.

Doing that also surfaced a decode bug. The AMSU-A/HSB blocks that follow the AIRS channels aren't always separated by another replication marker, so a few microwave channels were leaking through as AIRS rows. The decoder now uses the declared replication count as the block boundary.

@aayushg55

aayushg55 commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

@negin513 Thanks for the restructure.

I had my agent review the NNJA IR based on our current work/findings in converting the same bufr to parquet. It produced a rather lengthy document which I think has some highly relevant information which would be hard to distill otherwise, so attaching in its full here as documentation/reference. The summary and the suggested changes are both at the top, everything after that is the additional evidence.
e2studio_pr1046_nnja_ir.md

The four correctness items, repeated here:

  1. AIRS route. airsev covers 2007-02-21 → 2020-12-07, but airs/nasa/aqua in the same bucket
    covers a longer 2002-08-31 → 2023-01-18. The two are structurally identical and mostly agree on the infrared
    values. Worth defaulting to the NASA route, or exposing both somehow. (We use the NASA route)
  2. CrIS FORN is decoded into scalars and then never emitted. CrIS is the one sensor where
    FOVN is not the cross-track position — FOVN is 1–9 within the field of regard, FORN is 1–30
    across track, and all 270 pairs occur. So CrIS currently comes out of this source with no
    cross-track position at all, which is what edge-of-scan screening and scan-angle bias correction
    key on. jpss_cris.py already carries the two as separate columns.
  3. CrIS quality is hard-coded to "none". crisf4 carries five quality fields. NFQF is the one
    that flags the rows damaged by the NOAA-21 neon event, and those radiances are finite, so a
    consumer can't spot them any other way. The long layout makes this tractable — resolve the band
    for each channel and pack that band's flags.
  4. quality is uint16 but AIRS ACQF is a 24-bit flag table. A value above 65535 doesn't
    truncate, it raises, and that happens outside the per-message try, so one such value fails the
    whole request. uint32 covers it.

The rest is mainly nice to have code deduplication and some comments worth correcting.

There are also some points on performance/memory, but this is not really specific to this implementation but rather generally applicable for all these raw high-resolution MW/IR sounders data sources, and particularly for IR which has hundreds of channels. These would probably be best addressed in future PRs. TLDR, as is, 48hr of satellite data might consume a lot of memory.

- Route AIRS to airs/nasa/aqua (2002-2023) instead of airs/airsev
  (2007-2020); the NASA route is structurally identical and spans five
  more early years plus the instrument's final two. Adds a per-product
  naming convention since the NASA files are named
  airs_disc_final.YYYYMMDD.tHHz.bufr rather than gdas.*.tm00.bufr_d.
  Verified against the live bucket.
- Emit CrIS FORN (cross-track, 1-30) as scan_position; FOVN is the 3x3
  within-FoR detector index (1-9) and is not the cross-track position.
- Decode CrIS per-band NFQF/NCQF quality flags: resolve each channel's
  band and pack nfqf | ncqf << 19 into the quality column. NFQF is the
  flag that fires on real sensor failures (e.g. the NOAA-21 neon event)
  whose damaged radiances are otherwise finite and plausible.
- Widen quality to uint32: AIRS ACQF is a 24-bit flag table, and a value
  above 65535 raised ArrowInvalid outside the per-message try, failing
  the whole request.
- Drop C1/C2/brightness_temperature from utils_ir.py in favor of the
  shared radiance_to_bt / PLANCK_C1 / PLANCK_C2 in utils.py, already
  used by the MetOp and JPSS granule sources for the same instruments.
- Switch the shared inversion from np.log1p to np.log(1 + x): the
  argument never falls below ~16 over Earth scenes, so log1p buys no
  accuracy and is measurably slower.
- Correct the radiance_to_bt docstring, which claimed CODATA 2018 but
  printed the 2014 values.
- Add a cross-consistency test asserting wavenumber_cm_inverse("cris")
  matches jpss_cris._CRIS_WAVENUMBER_APOD element for element, and that
  the IASI channel count matches metop_iasi, so the grids cannot drift
  apart silently.
- Validate sensor_indices at construction (non-empty, unique, integer,
  within the instrument grid) so an empty or out-of-range selection
  raises instead of silently returning an empty frame after downloading
  a multi-GB aggregate; matches the JPSS/METOP sources' behavior.
- Recognize DRF8BIT (0-31-001) as the crisf4 guard-spectrum block
  boundary and scope the second-DRF16BIT break to AIRS, so the guard
  protection is explicit and a nested replication block in a future
  IASI/CrIS product cannot silently truncate footprints. The comment now
  states the channel budget is the primary terminator with markers as
  backstops.
- Drop footprints missing their cross-track position (FORN for CrIS,
  FOVN otherwise) instead of emitting None into the non-nullable
  scan_position column, matching the microwave path.
- Resolve IASI/CrIS channel wavenumbers once per footprint instead of
  once per channel, removing a one-element numpy allocation per channel
  from the hot loop; hoist the conversion imports to module scope.
- Log a debug count of IASI channels skipped for missing CHSF bands so
  a narrower-than-expected band table is distinguishable from no data.
- Documentation: elev is CrIS-only (SELV is platform altitude, not a
  substitute); scan_angle is always NaN for IR; per-sensor
  scan_position semantics; real per-cycle row magnitudes in the memory
  warning; CHSF is a channel-group exponent, not per-band.
Revert the ISD, IBTrACS, CFS reforecast, and OPERA obstore migrations to
their main state; they now land via NVIDIA#1058 so this PR is NNJA IR only.
@negin513 negin513 changed the title wip(data): add NNJAObsIRSat Add hyperspectral IR sounder variables (AIRS, IASI, CrIS) to NNJAObsSat Aug 12, 2026

@aayushg55 aayushg55 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @negin513 for following through with all of the changes!

- Correct the quality column metadata: the microwave path populates it
  from the per-channel 0-33-081 flags, so "MW: null" was wrong; only
  IASI leaves it null.
- Move the utils_ir module docstring into a comment block below the
  license header per e2s-002 (only license headers above imports).
- Raise on an uninitialized BUFR decoder worker in the IR batch fn,
  matching the microwave path, instead of miscounting it as per-message
  decode failures.
@negin513

Copy link
Copy Markdown
Member Author

/ok to test f4f0679

@negin513

Copy link
Copy Markdown
Member Author

Thanks @aayushg55 . All issues are addressed and started a perf PR on #1059 for vectorized conversion.

@negin513

Copy link
Copy Markdown
Member Author

Will bring this in once all tests pass.

@negin513
negin513 removed the request for review from NickGeneva August 12, 2026 22:15
@negin513

Copy link
Copy Markdown
Member Author

/ok to test cbfb1f6

@negin513
negin513 merged commit 505b394 into NVIDIA:main Aug 13, 2026
15 checks passed
@negin513
negin513 deleted the obstore-independent-four branch August 13, 2026 00:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants