Skip to content

S3 on Forge #6

Closed
5 of 5 issues completed
Closed

Description

@alanshaw

Track the prototype-to-production path for an S3-compatible API on top of the Forge stack. The S3 listener (codename ms3t) currently lives at pkg/ms3t/ in this repo. See pkg/ms3t/architectural.md for the prototype-level design notes, the choice points, and open questions.

Why sprue, not a separate repo? A pragmatic prototype-stage call: keep ms3t in-process with sprue so it inherits sprue's identities + clients with zero auth coordination. Up for re-evaluation once we settle a tenant model.

In scope for this epic

  • #37 — Define a sensible S3 API subset for the initial spike
  • #38 — Bootstrap S3 facade (HTTP listener + MST + Forge integration) — anchored on #34
  • #39 — Bucket mutation data storage: investigate "separate space for block diffs"
  • #40 — Add ms3t to smelt + S3 smoke tests
  • #41 — ms3t telemetry + Grafana dashboards
  • #42 — Stand up staging deployment + production infra prep (spike)
  • #43 — Flesh out tenant management API (spike)

Status snapshot

# Title Status
#7 Define S3 API subset for the spike proposed
#8 Bootstrap S3 facade in flight (#34)
#9 Bucket mutation data storage proposal proposed
fil-forge/smelt#16 ms3t in smelt + S3 smoke tests proposed
fil-forge/ingot#37 ms3t telemetry + Grafana dashboards partial (#35, storacha/smelt#23)
#12 Staging deployment + production infra (spike) proposed
#13 Tenant management API (spike) proposed

Explicitly deferred

  • Sharding. Assume all data received is <256 MB. The sharding proposal in flight is fil-one/RFC#2; revisit once it lands. Adjacent prior art: storacha/RFC#65 (Filepack archive format), storacha/RFC#66 (Virtual DAG in Sharded DAG Index), and shard.rfc in this repo.
  • Multipart upload, aws-chunked body decoding, AWS Sig V4 auth, pre-signed URLs, server-side encryption, x-amz-meta-* headers, versioning, object lock — these are FilOne hard requirements but are out of scope until #43's spike proposes a sequencing.
  • GC of unreferenced bodies, multi-instance / sticky-bucket routing, direct-passthrough to piri — see pkg/ms3t/architectural.md#future-directions.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions