Add ERA5 Copernicus CDS metforcing plugin

[hkato1]

Description

ERA5 data is publicly available in 0.25-deg gridded format from the Copernicus Climate Data Store. This implementation works with the dataset downloaded from CDS, "ERA5 hourly data on single levels from 1940 to present".
The LDT/LIS reader for ERA5 exists in metforcing/era5 but they are in different structure and not updated to present.
This reader is named "era5cds" and the ERA5 data is continuously updated daily with 5-day latency.

Resolves #1812

Testcase

/discover/nobackup/hkato/LIS/LIS7/era5cds/testcase

[emkemp]

Hi @hkato1

I can't find your landmask_mod44w_025.1gdr file needed to run your LDT test case. Please advise.

[dmocko]

@emkemp

I knew I had seen this file around before; here it is (same file in both directories):

/discover/nobackup/projects/lis/LS_PARAMETERS> find . -name landmask_mod44w_025.1gd4r -print 2>&1 | grep -v 'permission denied' | grep -v old

./clsmf25_parms/GLDAS_0.25-deg/landmask_mod44w_025.1gd4r
./MODIS/MOD44W.005/landmask_mod44w_025.1gd4r

[emkemp]

Thanks @dmocko!

[emkemp]

Okay, here's a new one: Where is era5cds_elev.nc?

[hkato1]

Hi! I followed the instruction by Jim on creating testcase. The input files for LDT should be included in era5cdsforcingtest_ldt.tar.gz.

[emkemp]

@hkato1 Ah, okay. Sorry, I was looking at the work directory itself. I'll work with the tar files. Thanks.

[hkato1]

Sorry, I forgot to mention them in README. I updated the README file.

[emkemp]

Update: Test output has been reproduced. I will now play a bit with the code.

[emkemp]

...aaand we have a bug.

../metforcing/era5cds/get_era5cds.F90(190): error #6784: The number of actual arguments cannot be greater than the number of dummy arguments. [ERA5CDSFILES]
call era5cdsfiles(n,kk,findex,era5cds_struc(n)%era5cdsdir, &
-------------^
../metforcing/era5cds/get_era5cds.F90(191): error #6633: The type of the actual argument differs from the type of the dummy argument. [HR1]
yr1, mo1, da1, hr1, instfilename, avgfilename)

The subroutine is defined as:

subroutine era5cdsfiles(n, kk, findex, era5cdsdir, yr, mo, da, fname)

So, an extra filename is being passed as an argument. era5cdsfiles needs to be fixed.

I'll continue looking at this tomorrow.

[hkato1]

oh, oops. I didn't update the routines in ldt/metforcing/era5cds after making latest changes to equivalent routines in lis/metforcing/era5cds. I don't know how it compiled without error when I tested. Thanks for catching that!

[emkemp]

Hi @hkato1

Your test case doesn't use the get_era5cds subroutine in LDT. Can you update your test case to do so?

BTW, I'm looking in /discover/nobackup/projects/lis/MET_FORCING/ERA5/single_hourly, and I don't see any time averaged files. The three file types I see are "accum", "instant", and "ml_instant". So I'm questioning how this is supposed to work.

[emkemp]

In addition, the LDT and LIS documentation need to be updated. Please edit:

ldt/configs/ldt.config.adoc
lis/configs/lis.config.adoc

[hkato1]

Hi @hkato1

Your test case doesn't use the get_era5cds subroutine in LDT. Can you update your test case to do so?

BTW, I'm looking in /discover/nobackup/projects/lis/MET_FORCING/ERA5/single_hourly, and I don't see any time averaged files. The three file types I see are "accum", "instant", and "ml_instant". So I'm questioning how this is supposed to work.

Hi @emkemp
Ah, that's why the LDT testcase didn't crash. I expect that I will need to update/modify the routines for the bug first. How does it work with the PR? Do I commit the changes in my branch then you can get updates?

It's not exactly time averaged, it's time accumulated. Sorry, it was confusing, I should clarify that in the routines. "accum" file contains the fields (i.e. precip, swdown, lwdown) that has different time stamps from "instant" and "ml_instant".

[emkemp]

Hi @hkato1

You can push additional changes to this PR--I have it marked as "Not Ready", so we won't approve it yet. I can then get the latest changes.

[hkato1]

Hi @emkemp

I just pushed the changes. I have a new LDT test case for using ERA5CDS in Metforce processing mode. Please see /discover/nobackup/hkato/LIS/LIS7/era5cds/testcase/ldt.config_met. The outputs are in TEST_OUTPUT/FORCING/194001. Also updated the original LDT testcase for LSM parameter processing in ldt.config following the update.

However, there are two problems.

1. The output of ldt.config.met contains only, lat, lon, and time. How can I add the meteorological fields? I checked another testcase using NLDAS2 met forcing, but it also outputs only lat, lon, and time.
2. As you can see in ldtlog.met.0000, there are many points with high SWdown field that are reset to solar constant value. This is an artifact of zenith angle computation and a fix needs to be inserted in zterp.F90. It is implemented and can be configured using "Enable new zterp correction (met forcing): .true." in LIS. LDT has slightly different structure and I am wondering how best to incorporate a modification in zterp routine.
3. LDT's Metforce processing is similar but not the same as how met forcing are processed in LIS? Is it ok if ERA5CDS in LDT doesn't have options supported such as ensemble, forecast, and "average" upscaling?

Please let me know what your thoughts are.
Thank you!
Hiroko

[emkemp]

Hi @hkato1

LIS will not compile with strict checks (-3). I get the following errors.

../metforcing/era5cds/get_era5cds.F90(211): error #6404: This name does not have a type, and must have an explicit type. [AVGFILENAME]
yr1, mo1, da1, hr1, instfilename, avgfilename, lmlfilename, &
-----------------------------------------------^
../metforcing/era5cds/get_era5cds.F90(212): error #6404: This name does not have a type, and must have an explicit type. [PREVAVGFILENAME]
prevavgfilename)
-------------^
../metforcing/era5cds/get_era5cds.F90(217): error #6099: An ENDDO statement occurred without a corresponding DO or DO WHILE statement.
enddo
^
../metforcing/era5cds/get_era5cds.F90(74): remark #7712: This variable has not been used. [R]
integer :: c, r,kk,f,try
--------------------------^
../metforcing/era5cds/get_era5cds.F90(74): remark #7712: This variable has not been used. [TRY]
integer :: c, r,kk,f,try
---------------------------------^
compilation aborted for ../metforcing/era5cds/get_era5cds.F90 (code 1)
make: *** [Makefile:129: get_era5cds.o] Error 1

[ERR] Compile failed

I'll look at your questions for LDT.

[hkato1]

Sorry about that! It's amazing it didn't catch it when I compiled with -1. Please try again.

[emkemp]

Hi @hkato1

LIS compiles with your latest changes, but it crashes on your test case. I just reran with strict checks, and I get this error:

forrtl: severe (408): fort: (2): Subscript #1 of the array METDATA1 has value 2 which is greater than the upper bound of 1

Image PC Routine Line Source
LIS 00000000080C18A4 assign_processed_ 1075 read_era5cds.F90
LIS 00000000081450A1 read_era5cds_ 776 read_era5cds.F90
LIS 00000000043DD5ED get_era5cds_ 214 get_era5cds.F90
LIS 0000000001ABBAA2 retrievemetforc_ 229 LIS_metforcing_FTable.c
LIS 0000000001A7C7E4 lis_metforcingmod 1090 LIS_metforcingMod.F90
LIS 00000000087A655E retrospective_run 218 retrospective_runMod.F90
LIS 00000000087A63D3 retrospective_run 129 retrospective_runMod.F90
LIS 0000000001BC18A1 lisrun_ 220 LIS_runmode_FTable.c
LIS 0000000004D580EE MAIN__ 70 lisdrv.F90
LIS 000000000041B40D Unknown Unknown Unknown

So I'm afraid more work is needed.

[hkato1]

Hi @emkemp

Thanks for pointing out the problem. I finally fixed it and committed the update. There was a section of routine that got copied over twice accidentally. I was able to run the testcase. Please try again.

[emkemp]

Thanks @hkato1. I succeeded in reproducing your LIS results.

Going back to your LDT metforcing results, your ldt_forcing_vars.txt is not formatted correctly. You only have four columns (plus the comment describing the variable). There are actually six columns -- column 5 indicates time averaging or accumulation option, and column 6 indicates this is a forcing variable. Column 6 must be set to 1 to be processed correctly. If a column is missing, it will use the default value (I believe 0).

Also, the units must be correct for the variable in question. You had:

Wind_E: 1 1 W/m2 # Eastward wind

That is an invalid unit for Wind_E. In my test version, I replaced with:

Wind_E: 1 1 m/s 0 1 # Eastward wind

And I get forcing data output. Look in /discover/nobackup/projects/usaf_lis/emkemp/pr/1813/work/TEST_OUTPUT/FORCING/194001

I request you rerun your test case with the corrected ldt_forcing_vars.txt.

[hkato1]

Hi @emkemp

Thanks for figuring out that my ldt_forcing_vars.txt was in wrong format. I was using forcing_variables.txt in LIS, which I had no idea that the contents were different. I was able to produce forcing data outputs, but examining the contents made me realize that the elevation correction was not working correctly. It took a while, but I finally fixed it and just committed the updates. In my testcase directory, you will see two sets of config file and outputs:
With elevation correction: ldt.config_met and outputs in TEST_OUTPUT/FORCING/194001
Without elevation correction: ldt.config_met_noecor and outputs in TEST_OUTPUT2/FORCING/194001.

As expected, these outputs match the forcing fields generated with template option in LIS (TEST_OUTPUT/SURFACEMODEL/ and TEST_OUTPUT2/SURFACEMODEL/ with lapse-rate correction and without it, respectively), except for the SWdown fields where I have new zterp correction in LIS. This goes back to the issue 2 that I mentioned earlier. Without the modification, SWdown fields are too high due to artificially large weights returned from zterp.

In LDT, zterp is turned on via "LDT_rc%met_zterp(findex) = .true." in era5cds_forcingMod.F90.
Should I create additional variable like LDT_rc%new_met_zterp to allow a switch for using modified zterp?

[emkemp]

Good morning @hkato1

Can you open up permissions for your latest directories:

/discover/nobackup/hkato/LIS/LIS7/era5cds/testcase/TEST_OUTPUT/FORCING
/discover/nobackup/hkato/LIS/LIS7/era5cds/testcase/TEST_OUTPUT2/FORCING

?

I have a meeting scheduled today with @jvgeiger to review PRs. We will discuss your question about LDT_rc%new_met_zterp.

[hkato1]

Good morning, @emkemp
The permissions should be open now. Sorry about that!

[emkemp]

Nope, not yet.

[emkemp@discover35 /discover/nobackup/hkato/LIS/LIS7/era5cds/testcase/TEST_OUTPUT]$ ls -l FORCING
ls: cannot open directory 'FORCING': Permission denied
[emkemp@discover35 /discover/nobackup/hkato/LIS/LIS7/era5cds/testcase/TEST_OUTPUT]$ ls -l | grep FORCING
drwx------ 4 hkato s1189 4096 Jun 22 11:54 FORCING/

[emkemp@discover35 /discover/nobackup/hkato/LIS/LIS7/era5cds/testcase/TEST_OUTPUT2]$ ls -l FORCING
ls: cannot open directory 'FORCING': Permission denied
[emkemp@discover35 /discover/nobackup/hkato/LIS/LIS7/era5cds/testcase/TEST_OUTPUT2]$ ls -l | grep FORCING
drwx------ 3 hkato s1189 4096 Jul 16 13:47 FORCING/

[hkato1]

Sorry!, Please try again.

[emkemp]

Hi @hkato1:

I can access your test output. Bad news: I cannot reproduce your TEST_OUTPUT results with ldt.config_met, even with
LDT_HIST_194001010800.d01.nc.

I used the intel_2023.2.1 compiler set. MY compile settings are in /discover/nobackup/projects/usaf_lis/emkemp/pr/1813/LISF/ldt/make/configure.ldt. Can you point me to your configure.ldt so I can compare?

[hkato1]

Hi @emkemp
My configure.ldt is here:
/discover/nobackup/hkato/LIS/LIS7/era5cds/src/ldt/make/configure.ldt
I am also using intel_2023.2.1 but it's dated Apr 18, 2025--perhaps I should re-create it by running configure?

[emkemp]

@hkato1

Do you mean your LDT executable is dated 18 Apr 2025? I'm expecting you generated your latest results with the same code as in this pull request.

[hkato1]

@emkemp
No, my configure.ldt was created in 2025. I haven't run configure since then.
LDT executable has date stamp of Jul 22.

[emkemp]

Okay, so an update:

I still cannot reproduce your LDT met test results. I went ahead and copied your source code in /discover/nobackup/hkato/LIS/LIS7/era5cds/src and compiled several times, most recently with the environment specified by /home/hkato1/privatemodules/lisf_7.5_intel_2023.2.1. Note that I'm using your configure.ldt, LIS_misc.h, and LDT_NetCDF_inc.h files.

At this point I'm guessing there's something in your .cshrc file that is causing this difference. These are not major differences, but I expect bitwise identical results. I will continue to investigate.

[hkato1]

Good morning, @emkemp
That's puzzling. Are you seeing the differences in swdown field? I reran ldt.config_met and ldt.config_met_noecor just to start fresh. Somehow, swdown is different between TEST_OUTPUT and TEST_OUTPUT_old (and TEST_OUTPUT2 and TEST_OUTPUT2_old), though the rest of fields are identical. I'm plotting the fields in GrADS and subtracting them to compare. Can you check against my new outputs in TEST_OUTPUT and TEST_OUTPUT2?
Thanks!

[emkemp]

Hi @hkato1

I can reproduce your latest TEST_OUTPUT results, but not TEST_OUTPUT2. For TEST_OUTPUT2, I see differences with SWdown_inst.

[emkemp]

Follow-up: When I run LDT compiled with strict checks (-3), I get a floating invalid error when running with ldt.config_met_noecor.

forrtl: error (65): floating invalid
Image PC Routine Line Source
libpthread-2.31.s 0000145716B30910 Unknown Unknown Unknown
LDT 0000000003D8FAE9 zterp_ 155 zterp.F90
LDT 0000000003B7BB66 timeinterp_era5cd 142 timeinterp_era5cds.F90
LDT 00000000014AF7A8 Unknown Unknown Unknown
LDT 000000000140ECF2 ldt_metforcingmod 851 LDT_metforcingMod.F90
LDT 00000000016C16F3 ldt_run_metforcpr 25 LDT_run_MetforcProc.F90
LDT 00000000016C3E67 Unknown Unknown Unknown
LDT 0000000001960395 MAIN__ 47 LDTmain.F90
LDT 000000000040A91D Unknown Unknown Unknown
libc-2.31.so 000014571309524D __libc_start_main Unknown Unknown
LDT 000000000040A84A Unknown Unknown Unknown

It seems more work is needed.

[hkato1]

Hi @emkemp

Some variables were not initialized in the timeinterp routine. This was the reason why we couldn't replicate identical swdown fields and swdown fields in LDT were not valid. I compiled both LDT and LIS with strict checks (-3) and re-run testcases in OUTPUT (with elevation correction) and OUTPUT2 (no elevation correction). I also re-run ldt.config for LSM parameter processing to make sure that the update didn't break it.

Were you able to discuss LDT_rc%new_met_zterp?

[emkemp]

Hi @hkato1

@jvgeiger and I believe you should proceed and add a new LVT_rc data member. But we recommend you use the same name (zterp_correction) that is used in LIS. Probably a similar ldt.config entry as well.

[hkato1]

Hi @emkemp

It turns out that "zterp_correction" was already available in LDT. I just had to turn it on in ldt.config. I added the modification in zterp in LDT, so the routine is consistent with LIS. Please see ldt.config_met_zterp_correction and TEST_OUTPUT in my testcase directory.
Thanks!