Skip to content

open_reader double-applies a store prefix= #744

Description

@blinkybool

On a store created with a prefix= (e.g. S3Store(prefix="P")), obstore.open_reader(store, "key") resolves the object at P/P/key instead of P/key, so every buffered read of a prefixed store fails with a 404. get, put, and list on the same store are correct (single prefix). Regressed in 0.9.0; still present in 0.11.0.

Environment

  • obstore 0.11.0 also reproduced on 0.9.0.
  • Last good: 0.8.2

Reproduction

Standalone uv run repro.py (deps declared inline; moto provides a real S3 backend). The object is written once at P/key; swap the obstore pin to 0.8.2 to see the correct behavior.

# /// script
# requires-python = ">=3.11"
# dependencies = ["obstore==0.11.0", "boto3==1.43.22", "moto[server]==5.2.1"]
# ///
import boto3, obstore
from moto.server import ThreadedMotoServer
from obstore.store import S3Store

server = ThreadedMotoServer(port=0); server.start()
host, port = server.get_host_and_port()
endpoint = f"http://{host}:{port}"

s3 = boto3.client("s3", endpoint_url=endpoint, region_name="us-east-1",
                  aws_access_key_id="x", aws_secret_access_key="x")
s3.create_bucket(Bucket="bucket")
s3.put_object(Bucket="bucket", Key="P/key", Body=b"hello")  # object lives at P/key

store = S3Store(bucket="bucket", prefix="P", endpoint=endpoint, region="us-east-1",
                access_key_id="x", secret_access_key="x",
                client_options={"allow_http": True}, virtual_hosted_style_request=False)

print("list :", [o["path"] for b in obstore.list(store) for o in b])  # ['key']
print("head :", obstore.head(store, "key")["path"])                    # 'P/key'  <- should be 'key'
print("get  :", bytes(obstore.get(store, "key").bytes()))              # b'hello'
try:
    print("read :", obstore.open_reader(store, "key").read())          # -> GET P/P/key -> 404
except Exception as e:
    print("read :", str(e).splitlines()[0])
server.stop()

Output

obstore 0.11.0 (bug) — moto also logs HEAD /bucket/P/key then GET /bucket/P/P/key:

list : ['key']       # prefix stripped (correct)
head : P/key         # <- prefix NOT stripped (root cause)
get  : b'hello'      # correct
read : Object at location P/P/key not found: ... GET .../bucket/P/P/key ... 404

obstore 0.8.2 (correct) — GET goes to /bucket/P/key:

list : ['key']
head : key
get  : b'hello'
read : Bytes(b'hello')

Expected

open_reader(store, "key") on a store with prefix="P" should read P/key (as get/put/list do), and head(...)["path"] should return "key" (prefix-relative), consistent with list().

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions