On a store created with a prefix= (e.g. S3Store(prefix="P")), obstore.open_reader(store, "key") resolves the object at P/P/key instead of P/key, so every buffered read of a prefixed store fails with a 404. get, put, and list on the same store are correct (single prefix). Regressed in 0.9.0; still present in 0.11.0.
Environment
- obstore 0.11.0 also reproduced on 0.9.0.
- Last good: 0.8.2
Reproduction
Standalone uv run repro.py (deps declared inline; moto provides a real S3 backend). The object is written once at P/key; swap the obstore pin to 0.8.2 to see the correct behavior.
# /// script
# requires-python = ">=3.11"
# dependencies = ["obstore==0.11.0", "boto3==1.43.22", "moto[server]==5.2.1"]
# ///
import boto3, obstore
from moto.server import ThreadedMotoServer
from obstore.store import S3Store
server = ThreadedMotoServer(port=0); server.start()
host, port = server.get_host_and_port()
endpoint = f"http://{host}:{port}"
s3 = boto3.client("s3", endpoint_url=endpoint, region_name="us-east-1",
aws_access_key_id="x", aws_secret_access_key="x")
s3.create_bucket(Bucket="bucket")
s3.put_object(Bucket="bucket", Key="P/key", Body=b"hello") # object lives at P/key
store = S3Store(bucket="bucket", prefix="P", endpoint=endpoint, region="us-east-1",
access_key_id="x", secret_access_key="x",
client_options={"allow_http": True}, virtual_hosted_style_request=False)
print("list :", [o["path"] for b in obstore.list(store) for o in b]) # ['key']
print("head :", obstore.head(store, "key")["path"]) # 'P/key' <- should be 'key'
print("get :", bytes(obstore.get(store, "key").bytes())) # b'hello'
try:
print("read :", obstore.open_reader(store, "key").read()) # -> GET P/P/key -> 404
except Exception as e:
print("read :", str(e).splitlines()[0])
server.stop()
Output
obstore 0.11.0 (bug) — moto also logs HEAD /bucket/P/key then GET /bucket/P/P/key:
list : ['key'] # prefix stripped (correct)
head : P/key # <- prefix NOT stripped (root cause)
get : b'hello' # correct
read : Object at location P/P/key not found: ... GET .../bucket/P/P/key ... 404
obstore 0.8.2 (correct) — GET goes to /bucket/P/key:
list : ['key']
head : key
get : b'hello'
read : Bytes(b'hello')
Expected
open_reader(store, "key") on a store with prefix="P" should read P/key (as get/put/list do), and head(...)["path"] should return "key" (prefix-relative), consistent with list().
On a store created with a
prefix=(e.g.S3Store(prefix="P")),obstore.open_reader(store, "key")resolves the object atP/P/keyinstead ofP/key, so every buffered read of a prefixed store fails with a 404.get,put, andliston the same store are correct (single prefix). Regressed in 0.9.0; still present in 0.11.0.Environment
Reproduction
Standalone
uv run repro.py(deps declared inline; moto provides a real S3 backend). The object is written once atP/key; swap theobstorepin to0.8.2to see the correct behavior.Output
obstore 0.11.0 (bug) — moto also logs
HEAD /bucket/P/keythenGET /bucket/P/P/key:obstore 0.8.2 (correct) — GET goes to
/bucket/P/key:Expected
open_reader(store, "key")on a store withprefix="P"should readP/key(asget/put/listdo), andhead(...)["path"]should return"key"(prefix-relative), consistent withlist().