Discussion about private direct bucket data access

[kylebarron]

Some options:

• server data proxy. This isn't ideal because the server needs to proxy all data, so this puts a lot of egress load onto the server.
• signed urls: this isn't ideal for Zarr because normally Zarr is stored on S3, and you would have to sign every url independently
• STS tokens: AWS supports generating short term AWS credentials scoped to specific bucket operations and optionally a bucket prefix. We could use these tokens to generate signed AWS requests from the client, so the client would only need to make one server request, and the rest of the data could be fetched directly from object storage.
  • I succeeded at getting obstore compiled to wasm (got a vibe coded prototype of this working, so it could work in theory)

[kylebarron]

There are a bunch of ways to do this, which tend to be specific to the backend needs of any given project.

At the point where this is prioritized, we plan to create a TypeScript abstraction library, lightly in the design of obstore that uses STS tokens in the client to do HEAD, GET, and LIST requests on S3.