Skip to content

feat: refactor from_source handling - #1169

Merged
sandorkertesz merged 19 commits into
developfrom
refactor/source_handling
Nov 3, 2026
Merged

sandorkertesz merged 19 commits into
developfrom
refactor/source_handling

Conversation

@Oisin-M

@Oisin-M Oisin-M commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Description

Related to #1097 #162. Diff is quite large, so apologies for that...

PR aims to refactor and simplify the from_source handling, and to allow us to properly type things. Everything is backwards compatible for now, but in future we could easily drop **kwargs from _from_file etc. and have things typed properly.

Steps

  • Implement refactor of from_source
  • Add properly typed _from_file etc.
  • Remove existing SourceMaker, SourceLoader, _from_source etc. since they will no longer be required
  • Remove support for "private" sources i.e. from_source("dummy-source" etc.. Tests now use _from_dummy_source, but users/code agents etc. can no longer "discover" hidden sources

(subsequent PR: #1170)

  • Refactor reader handling
  • Add plugin system for readers via "earthkit.data.readers" where should just link to a matching function def reader(source, path, *, magic=None, deeper_check=False, content_type=None, **kwargs):

Tradeoff:
This does mean that now adding new sources in earthkit-data is a little more work. You would need to add the source as before, but then also add a _from_new_source method (so we know the valid kwargs!), and then do the (one-line) plumbing to the main from_source method. I think this is worth the effort.

Contributor Declaration

By opening this pull request, I affirm the following:

  • All authors agree to the Contributor License Agreement.
  • The code follows the project's coding standards.
  • I have performed self-review and added comments where needed.
  • I have added or updated tests to verify that my changes are effective and functional.
  • I have run all existing tests and confirmed they pass.

@Oisin-M
Oisin-M marked this pull request as ready for review September 24, 2026 14:08
@Oisin-M
Oisin-M added this pull request to stack #1171 September 25, 2026 08:22
@Oisin-M
Oisin-M force-pushed the refactor/source_handling branch from 0c3efac to 73e23fd Compare September 29, 2026 11:46
@Oisin-M

Oisin-M commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

Failing tests are unrelated to this PR and also failing on current develop.

@sandorkertesz
sandorkertesz merged commit 23e4d88 into develop Nov 3, 2026
116 of 143 checks passed
@Oisin-M
Oisin-M deleted the refactor/source_handling branch November 3, 2026 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants