Skip to content

Migrate to earthkit v1.0 release candidate #162

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
frazane wants to merge 25 commits into
base: main
Choose a base branch
from
Open

Migrate to earthkit v1.0 release candidate #162

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
0c71de5
chore: run snakefmt
frazane May 26, 2026
4475d1f
update main project dependencies
frazane May 26, 2026
b440427
update dependencies for inference environment
frazane May 26, 2026
d2c4f2a
renaming dimensions
frazane May 26, 2026
6107544
refactor source files for new earthkit
frazane May 26, 2026
f6c0fce
add earthkit workaround instructions
frazane May 26, 2026
b7e547b
wip: update inference configs
frazane May 26, 2026
f4a972d
remove unused inference configs
frazane May 26, 2026
563d695
update configs with patches
frazane May 26, 2026
45e562b
Merge branch 'main' into feat/earthkit-v1-migration
frazane May 26, 2026
ad54000
standardize lat/lon coordinate names to latitude/longitude
frazane May 26, 2026
9663648
rename lead time dimension
frazane May 27, 2026
f3d033e
update config
frazane May 27, 2026
b5014ed
bugfix, use grib store
frazane May 28, 2026
b0d0818
update showcase code
frazane May 28, 2026
d894d86
fix colormap tests for list-of-colors loader contract
frazane May 28, 2026
4a211c2
merge origin/main into feat/earthkit-v1-migration
frazane May 28, 2026
d77bead
bugfixes after merge conflicts
frazane May 28, 2026
d568c40
fix wrong paramId for dewpoint temperature
frazane May 28, 2026
a186804
more fixes to wrong paramId
frazane May 28, 2026
19d4302
bugfix: compute uv components in obs dataset
frazane May 28, 2026
a92425d
consider both possible formats for step
frazane May 28, 2026
76b486c
remove pinned versions from config
frazane May 29, 2026
75301d6
pin anemoi inference to current main HEAD
frazane May 29, 2026
b201c48
do not set short lived var
frazane May 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -276,3 +276,17 @@ All outputs are rooted at `OUT_ROOT` (from `locations.output_root` in the config
| `{param}` | variable name | `T_2M`, `TOT_PREC` |
| `{region}` | geographic region slug | `switzerland`, `globe` |
| `{leadtime}` | zero-padded hours | `000`, `006`, `024` |


## earthkit v1.0rc support

The adoption of the first upcoming stable release of earthkit has started.
Some issues may still be present. A known issue that is affecting us is that the automatic download and caching of files required
by earthkit to support the ICON-CH grids is currently broken. This requires us to perform this step manually.

```bash
export ECKIT_GEO_CACHE=$HOME/.local/share/eckit/geo/grid/icon
mkdir -p $ECKIT_GEO_CACHE
curl https:/sites.ecmwf.int/repository/eckit/geo/grid/icon-ch/icon-ch1-c.ek > $ECKIT_GEO_CACHE/17643da2574959b644d254a3cd6e2bc0-b0699f374c63d05028c18c12f80a48f4.ek
curl https:/sites.ecmwf.int/repository/eckit/geo/grid/icon-ch/icon-ch2-c.ek > $ECKIT_GEO_CACHE/bbbd5a09855499243c7a4aa4c8762920-67adabf5c0cff041ebaafa61a3bda267.ek
```
2 changes: 1 addition & 1 deletion config/forecasters-ich1-oper-fixed.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ runs:
steps: 0/120/6
config: resources/inference/configs/sgm-multidataset-forecaster-global-ich1-oper.yaml
extra_requirements:
- git+https://github.com/ecmwf/anemoi-inference.git@b9aaee5df86614cad9d8d08b76876a4be4e980db
- git+https://github.com/ecmwf/anemoi-inference.git@de4cc0decd47dd593d2bb04cf9247e57a216d3cd

# - forecaster:
# checkpoint: https://service.meteoswiss.ch/mlstore#/experiments/602/runs/233b1274e79b4bfea98378d196a8aba0
Expand Down
2 changes: 1 addition & 1 deletion config/forecasters-ich1-oper.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ runs:
extra_requirements:
- earthkit-utils<0.2.0
- earthkit-data<0.19.0
- git+https://github.com/ecmwf/anemoi-inference.git@main
- git+https://github.com/ecmwf/anemoi-inference.git@de4cc0decd47dd593d2bb04cf9247e57a216d3cd

- baseline:
label: ICON-CH1-ctrl
Expand Down
4 changes: 2 additions & 2 deletions config/forecasters-ich1.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ runs:
extra_requirements:
- earthkit-utils<0.2.0
- earthkit-data<0.19.0
- git+https://github.com/ecmwf/anemoi-inference.git@main
- git+https://github.com/ecmwf/anemoi-inference.git@de4cc0decd47dd593d2bb04cf9247e57a216d3cd

- forecaster:
checkpoint: /scratch/mch/apennino/output/checkpoint/973b78b8b39543949abc2b154c5f98d9/inference-last.ckpt
Expand All @@ -26,7 +26,7 @@ runs:
extra_requirements:
- earthkit-utils<0.2.0
- earthkit-data<0.19.0
- git+https://github.com/ecmwf/anemoi-inference.git@main
- git+https://github.com/ecmwf/anemoi-inference.git@de4cc0decd47dd593d2bb04cf9247e57a216d3cd

# - forecaster:
# checkpoint: https://huggingface.co/met-no/bris_cloudy-skies/blob/main/2025-10/cloudy-skies-2025-10.ckpt
Expand Down
4 changes: 0 additions & 4 deletions config/interpolators-ich1.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,14 +23,10 @@ runs:
steps: 0/120/6
extra_requirements:
- git+https://github.com/ecmwf/anemoi-inference.git@e369b1a90313e9701db13f63364a467aa281cf36
- eccodes==2.39.1
- eccodes-cosmo-resources-python==2.38.3.1
extra_requirements:
- git+https://github.com/ecmwf/anemoi-inference.git@e369b1a90313e9701db13f63364a467aa281cf36
# pinned anemoi-datasets because of ecmwf/anemoi-utils#284, can be removed when fixed
- anemoi-datasets==0.5.35
- eccodes==2.39.1
- eccodes-cosmo-resources-python==2.38.3.1
- baseline:
label: INCA
root: /store_new/mch/msclim/INCA
Expand Down
18 changes: 8 additions & 10 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ dependencies = [
"snakemake<9.10", # https://github.com/snakemake/snakemake/issues/3289
"snakemake-executor-plugin-slurm",
"click",
"meteodata-lab>=0.4.0",
"anemoi-datasets>=0.5.31",
"mlflow>=3.1.1",
"pydantic>=2.11.7",
Expand All @@ -21,20 +20,18 @@ dependencies = [
"shapely>=2.1.2",
"cartopy>=0.25.0",
"pyproj>=3.7.2",
"earthkit-plots",
"marimo>=0.23.3",
"geopandas>=0.14.0",
"peakweather",
"pyzmq>=27.1.0",
"scores>=2.0.0",
]

[project.optional-dependencies]
kerchunk = [
"kerchunk",
"zarr<3.0.0",
"fastparquet",
"ujson"
"eccodes>=2.44,<2.48",
"eccodes-cosmo-resources-python==2.44.0.1",
"earthkit-data==1.0.0rc10",
"earthkit-utils==1.0.0rc1",
"earthkit-plots==1.0.0rc9",
"earthkit-meteo==1.0.0rc3",
"earthkit-geo==1.0.0rc7"
]

[project.scripts]
Expand Down Expand Up @@ -64,3 +61,4 @@ packages = [

[tool.uv.sources]
peakweather = { git = "https://github.com/MeteoSwiss/PeakWeather.git" }
eccodes-cosmo-resources-python = { git = "https://github.com/MeteoSwiss/eccodes-cosmo-resources-python" }
30 changes: 30 additions & 0 deletions resources/inference/configs/sgm-forecaster-global-disentangled.yaml
View file
Open in desktop
Comment thread
frazane marked this conversation as resolved.
Original file line number Diff line number Diff line change
Expand Up @@ -41,5 +41,35 @@ output:
inverse: true
- assign_mask: # fill local/global overlapping points with nan
mask: "global/cutout_mask"
modifiers:
- patches:
- variable:
U_10M: {"param": 165, "shortName": "10u"}
10u: {"param": 165, "shortName": "10u"}
V_10M: {"param": 166, "shortName": "10v"}
10v: {"param": 166, "shortName": "10v"}
TD_2M: {"param": 168, "shortName": "2d"}
2d: {"param": 168, "shortName": "2d"}
T_2M: {"param": 167, "shortName": "2t"}
2t: {"param": 167, "shortName": "2t"}
FR_LAND: {"param": 172, "shortName": "lsm"}
lsm: {"param": 172, "shortName": "lsm"}
PMSL: {"param": 151, "shortName": "msl"}
msl: {"param": 151, "shortName": "msl"}
PS: {"param": 134, "shortName": "sp"}
sp: {"param": 134, "shortName": "sp"}
SSO_SIGMA: {"param": 163, "shortName": "slor"}
slor: {"param": 163, "shortName": "slor"}
SSO_STDH: {"param": 160, "shortName": "sdor"}
sdor: {"param": 160, "shortName": "sdor"}
TOT_PREC: {"param": 228, "shortName": "tp"}
tp: {"param": 228, "shortName": "tp"}
z: {"param": 129, "shortName": "z"}
Comment thread
frazane marked this conversation as resolved.
"^q_(\\d+)$": {"param": 133, "shortName": "q"}
"^t_(\\d+)$": {"param": 130, "shortName": "t"}
"^u_(\\d+)$": {"param": 131, "shortName": "u"}
"^v_(\\d+)$": {"param": 132, "shortName": "v"}
"^w_(\\d+)$": {"param": 135, "shortName": "w"}
"^z_(\\d+)$": {"param": 129, "shortName": "z"}

write_initial_state: true
32 changes: 32 additions & 0 deletions resources/inference/configs/sgm-forecaster-global-ich1-oper.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,38 @@ output:
encoding:
typeOfGeneratingProcess: 2
centre: ecmf
modifiers:
- patches:
- variable:
U_10M: {"param": 165, "shortName": "10u"}
10u: {"param": 165, "shortName": "10u"}
V_10M: {"param": 166, "shortName": "10v"}
10v: {"param": 166, "shortName": "10v"}
TD_2M: {"param": 168, "shortName": "2d"}
2d: {"param": 168, "shortName": "2d"}
T_2M: {"param": 167, "shortName": "2t"}
2t: {"param": 167, "shortName": "2t"}
FR_LAND: {"param": 172, "shortName": "lsm"}
lsm: {"param": 172, "shortName": "lsm"}
PMSL: {"param": 151, "shortName": "msl"}
msl: {"param": 151, "shortName": "msl"}
PS: {"param": 134, "shortName": "sp"}
sp: {"param": 134, "shortName": "sp"}
SSO_SIGMA: {"param": 163, "shortName": "slor"}
slor: {"param": 163, "shortName": "slor"}
SSO_STDH: {"param": 160, "shortName": "sdor"}
sdor: {"param": 160, "shortName": "sdor"}
TOT_PREC: {"param": 228, "shortName": "tp"}
tp: {"param": 228, "shortName": "tp"}
z: {"param": 129, "shortName": "z"}
Comment thread
frazane marked this conversation as resolved.
"^q_(\\d+)$": {"param": 133, "shortName": "q"}
"^t_(\\d+)$": {"param": 130, "shortName": "t"}
"^u_(\\d+)$": {"param": 131, "shortName": "u"}
"^v_(\\d+)$": {"param": 132, "shortName": "v"}
"^w_(\\d+)$": {"param": 135, "shortName": "w"}
"^z_(\\d+)$": {"param": 129, "shortName": "z"}


templates:
samples: resources/templates_index_ifs.yaml
post_processors:
Expand Down
30 changes: 30 additions & 0 deletions resources/inference/configs/sgm-forecaster-global-ich1.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,5 +46,35 @@ output:
inverse: true
- assign_mask: # fill local/global overlapping points with nan
mask: "global/cutout_mask"
modifiers:
- patches:
- variable:
U_10M: {"param": 165, "shortName": "10u"}
10u: {"param": 165, "shortName": "10u"}
V_10M: {"param": 166, "shortName": "10v"}
10v: {"param": 166, "shortName": "10v"}
TD_2M: {"param": 168, "shortName": "2d"}
2d: {"param": 168, "shortName": "2d"}
T_2M: {"param": 167, "shortName": "2t"}
2t: {"param": 167, "shortName": "2t"}
FR_LAND: {"param": 172, "shortName": "lsm"}
lsm: {"param": 172, "shortName": "lsm"}
PMSL: {"param": 151, "shortName": "msl"}
msl: {"param": 151, "shortName": "msl"}
PS: {"param": 134, "shortName": "sp"}
sp: {"param": 134, "shortName": "sp"}
SSO_SIGMA: {"param": 163, "shortName": "slor"}
slor: {"param": 163, "shortName": "slor"}
SSO_STDH: {"param": 160, "shortName": "sdor"}
sdor: {"param": 160, "shortName": "sdor"}
TOT_PREC: {"param": 228, "shortName": "tp"}
tp: {"param": 228, "shortName": "tp"}
z: {"param": 129, "shortName": "z"}
Comment thread
frazane marked this conversation as resolved.
"^q_(\\d+)$": {"param": 133, "shortName": "q"}
"^t_(\\d+)$": {"param": 130, "shortName": "t"}
"^u_(\\d+)$": {"param": 131, "shortName": "u"}
"^v_(\\d+)$": {"param": 132, "shortName": "v"}
"^w_(\\d+)$": {"param": 135, "shortName": "w"}
"^z_(\\d+)$": {"param": 129, "shortName": "z"}

patch_metadata: resources/sgm-ich1-patch_metadata.yaml
30 changes: 30 additions & 0 deletions resources/inference/configs/sgm-forecaster-global.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,5 +39,35 @@ output:
inverse: true
- assign_mask: # fill local/global overlapping points with nan
mask: "global/cutout_mask"
modifiers:
- patches:
- variable:
U_10M: {"param": 165, "shortName": "10u"}
10u: {"param": 165, "shortName": "10u"}
V_10M: {"param": 166, "shortName": "10v"}
10v: {"param": 166, "shortName": "10v"}
TD_2M: {"param": 168, "shortName": "2d"}
2d: {"param": 168, "shortName": "2d"}
T_2M: {"param": 167, "shortName": "2t"}
2t: {"param": 167, "shortName": "2t"}
FR_LAND: {"param": 172, "shortName": "lsm"}
lsm: {"param": 172, "shortName": "lsm"}
PMSL: {"param": 151, "shortName": "msl"}
msl: {"param": 151, "shortName": "msl"}
PS: {"param": 134, "shortName": "sp"}
sp: {"param": 134, "shortName": "sp"}
SSO_SIGMA: {"param": 163, "shortName": "slor"}
slor: {"param": 163, "shortName": "slor"}
SSO_STDH: {"param": 160, "shortName": "sdor"}
sdor: {"param": 160, "shortName": "sdor"}
TOT_PREC: {"param": 228, "shortName": "tp"}
tp: {"param": 228, "shortName": "tp"}
z: {"param": 129, "shortName": "z"}
Comment thread
frazane marked this conversation as resolved.
"^q_(\\d+)$": {"param": 133, "shortName": "q"}
"^t_(\\d+)$": {"param": 130, "shortName": "t"}
"^u_(\\d+)$": {"param": 131, "shortName": "u"}
"^v_(\\d+)$": {"param": 132, "shortName": "v"}
"^w_(\\d+)$": {"param": 135, "shortName": "w"}
"^z_(\\d+)$": {"param": 129, "shortName": "z"}

write_initial_state: true
53 changes: 0 additions & 53 deletions resources/inference/configs/sgm-forecaster-global_trimedge.yaml
View file
Open in desktop

This file was deleted.

31 changes: 0 additions & 31 deletions resources/inference/configs/sgm-forecaster-regional.yaml
View file
Open in desktop

This file was deleted.

30 changes: 0 additions & 30 deletions resources/inference/configs/sgm-forecaster-regional_fromtraining.yaml
View file
Open in desktop

This file was deleted.

Loading
Loading