Add complevel argument to netcdf exporter + tests#550
Conversation
Setting this to a lower value significantly speeds up netcdf writing
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #550 +/- ##
=======================================
Coverage 84.17% 84.17%
=======================================
Files 170 170
Lines 14985 14985
=======================================
Hits 12614 12614
Misses 2371 2371
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
To complement @ladc pr, when using compression Level 9 the bottleneck when exporting is the compression leading to slow netcdf writes. Lowering the compression level only slightly increases output size but improves overall execution speed. This would help overall speed up without having to rely on other parallelization methods. Another change we could implement is to be able to rely on other compression libraries than zlib. |
Test whole range of compression levels (0-9).
larsbonnefoy
left a comment
There was a problem hiding this comment.
Adding simple new argument to exporter + test, looks got for me
There was a problem hiding this comment.
Pull request overview
Adds a complevel argument to initialize_forecast_exporter_netcdf so users can trade off netCDF compression ratio for write speed (the previous behavior hard-coded complevel=9). The default is preserved at 9 for backward compatibility, and the test matrix is extended with several compression-level values.
Changes:
- Add
complevelparameter (default 9) toinitialize_forecast_exporter_netcdfand forward it to bothcreateVariablecalls. - Document the new parameter in the docstring.
- Extend
test_io_export_netcdf_one_member_one_time_stepparametrization with acompleveldimension.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| pysteps/io/exporters.py | Replaces hard-coded complevel=9 with a user-configurable argument. |
| pysteps/tests/test_exporters.py | Adds complevel to the parametrized test cases (but does not pass it through to the exporter). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
As pointed out by @larsbonnefoy, setting this to a lower value significantly speeds up netcdf writing.