Conversation
HealDA-v2 NNJA training reads atmospheric motion vectors from the dedicated NCEP satwnd dump and drops the AMVs merged into PrepBUFR; earth2studio had no way to read that dump, so inference could only feed the PrepBUFR AMVs the model never saw. Add a data source for amv/satwnd in the NNJA archive (1979-present) that decodes every producer layout by resolving mnemonics from each file's embedded Table B and reading first occurrences the way GSI ufbint does. The GSI report type is not in the file. Port read_satwnd.f90's sattabin table so (subset, SAID, SWCM) maps to the same 240-260 type and processing case GSI assigns, resolve QI/EE per producer the way each GSI case reads them, and keep SWQM, SAZA, SWCM, height method and subset in extra columns. Winds GSI does not type are dropped; QC and thinning are left to the consumer.
The AMV dump only reports a pressure height assignment, so `NNJAObsSatwnd` no longer fills `elev` from the standard atmosphere. HealDA needs a geometric height for every conventional row, so `pressure_to_height_m` moves to `earth2studio.models.da.utils` and `HealDA.prep_conv` fills missing heights from pressure. Also drops the internal training-archive reference from the source docstring.
Draft
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Earth2Studio Pull Request
Description
Roadmap task: NVIDIA/physicsnemo-roadmap#2986 (feature NVIDIA/physicsnemo-roadmap#2669, Releasing HealDA V2 in E2S).
HealDA-v2 is trained on atmospheric motion vectors from the dedicated NCEP
satwnddump rather than the AMVs merged into PrepBUFR. Earth2Studio had no reader for that dump, so the only AMVs available at inference were the PrepBUFR ones inNNJAObsConv(report types 240-260), which the model never saw.This PR adds
NNJAObsSatwnd, aDataFrameSourceovers3://noaa-reanalyses-pds/observations/reanalysis/amv/satwnd/(1979-present, 6-hourly cycle files, 4-400 MB each), andearth2studio.data.utils_satwnd:NC0050xx, GOES-RNC00503x/05x, MeteosatNC00506x, HimawariNC00504x, MODIS/AVHRR/VIIRS, LEO-GEONC005072, INSAT) and NCEP-local descriptor ids drift between Table B versions (e.g.EHAMis 2162 in 2024 files while 2164 wasTCMDin 2015). Fields are resolved by mnemonic from each file's embedded Table B and read as first occurrence, the same as GSIufbint.TYP.SATWND_TYPE_TABLEportsread_satwnd.f90'ssattabin(NOAA-EMC/GSI860d1374):(NC005 subset, SAID, SWCM) -> (report type 240-260, processing case). Winds GSI does not type (SAIDs outside its platform lists, e.g. INSAT-3DR 473; unknown subsets) are dropped, as GSI drops them.qi,qi_forecast,expected_error.NCEP_CONVENTIONAL_PUBLIC_SCHEMAplussatellite_id,subset,wind_method(SWCM),height_method(HAMD/EHAM),satellite_za(signed),qi,qi_forecast,expected_error,gsi_case.u/vfromWDIR/WSPD;presis the final height assignment (Pa);elevis null (the dump carries no geometric height;HealDA.prep_convfills missing conventional heights from pressure viaearth2studio.models.da.utils.pressure_to_height_m, USSA-1976);qualityisSWQMwhere the producer encodes it;stationis GSI'sc_station_id(IR257,CT172, ...);classisSATWND.Not in this PR, deliberately: GSI-style QC screens (125 hPa floor, GEO zenith > 68 deg, SWCM >= 4 layer winds, speed caps) and the hpx5 / pressure-level / 2 h thinning that training applied. Those are model-side and tracked on the roadmap task, together with dropping PrepBUFR 240-260 in
NNJAObsConvwhen this source is used. The source exposes every field as the dump carries it, with no QC applied.Not verified: parity with the training archive for subsets GSI itself does not type (its ETL applied fallbacks that are not in
read_satwnd.f90).Real-file checks:
gdas.20150101.t00z.satwnd.tm00.bufr_d(35 MB): 275,822 winds across 11 subsets, every one typed as expected (245/246/251 GOES, 252/242/250 MTSAT, 253/243/254 Meteosat, 257/258/259 MODIS), QI resolved for all,SWQMpresent only for NESDIS/MODIS; 27 s with 8 decode workers vs 125 s single-worker.gdas.20240101.t00z.satwnd.tm00.bufr_d(294 MB): 3,382,615 winds across 15 subsets (GOES-16/18 240/245/246/247/251, Himawari-9 242/250/252, Meteosat-10/11 243/253/254, LEO-GEO 255, Metop AVHRR 244, NOAA-20/21 VIIRS 260); INSAT-3DR (NC005024/026, SAID 473) dropped as GSI drops it; QI resolved for 98% of winds (the remainder are GOES-R rows whose GNAPS 5 is missing); 177 s with 8 workers.Tests:
test/data/test_utils_satwnd.pycovers the type table againstsattabinspot checks, mnemonic resolution with file-table precedence, section-1 subcategory for editions 3 and 4, USSA inversion, legacy NESDIS / GOES-R / EUMETSAT-310077 / JMA subset layouts, time-window and missing-wind rejection, the end-to-enddecode_satwndpath with a mocked decoder, andNNJAObsSatwndURI/task planning.Checklist
Dependencies
None (uses the existing
dataextra:pybufrkit,obstore). TouchesHealDA.prep_convalongside #1150; the two merge cleanly (verified locally), only the CHANGELOG bullets are adjacent.