Skip to content

feat: Handle all forms of range requests - #766

Open
gschulze wants to merge 3 commits into
developmentseed:mainfrom
gschulze:feature/fsspec-range-requests
Open

feat: Handle all forms of range requests#766
gschulze wants to merge 3 commits into
developmentseed:mainfrom
gschulze:feature/fsspec-range-requests

Conversation

@gschulze

Copy link
Copy Markdown

Closes #259.

Adds support for all forms of range requests documented by fsspec.

Implemented:

  • get_range/get_ranges cover all three GetRange variants: bounded, start alone (to the end of the object), and negative start (the last N bytes). get_ranges takes None per element in ends/lengths, and resolves open-ended ranges with one head request so they still coalesce.
  • cat_file accepts either bound alone, or either counting back from the end. cat_ranges accepts fsspec's full vocabulary and now honors max_gap, which was previously accepted and silently ignored. test_cat_ranges_mixed is no longer xfail.

API changes:

  • Python:
    • ends/lengths: Sequence[int] | NoneSequence[int | None] | None.
    • coalesce: intint | None, where None means obstore's own default, previously hardcoded in four places across _get.pyi and store.py.
    • start stays int, but negative values are now accepted.
  • Rust:
    • start/starts: u64i64 on all four get_range*/get_ranges*, to support suffix requests.
    • ends/lengths: Option<Vec<u64>>Option<Vec<Option<u64>>>.
    • coalesce: u64Option<u64>.

Behavior changes:

get_ranges(starts=[5]) reads to the end of the object instead of raising; ends and lengths may both be passed if at most one is non-None per range; mismatched sequence lengths raise ValueError instead of being silently truncated.

Follow-up tasks:

Raise upstream: obstore now accepts range forms obspec's GetRange/GetRanges docstrings declare invalid. Conformance is unaffected, but code using the new forms won't port to other obspec implementations.

@ds-release-bot ds-release-bot Bot added the feat label Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fsspec: Handle all forms of range requests

1 participant