Skip to content

Bucket mutation data storage: investigate separate space for block diffs #9

Closed

Description

@alanshaw

Today, in storacha/sprue#34 every S3 PUT/DELETE produces a CAR carrying the body chunks, the new ObjectManifest, and the mutated MST path nodes. The bucket's "current root" is held in a small SQLite registry (pkg/ms3t/registry/). There is no separate persistent record of "the sequence of bucket mutations" — the MST history is implicit in the chain of CARs in Forge, but there's no separate index over it.

What's worth investigating

  • A separate space dedicated to bucket mutation block diffs, distinct from the body-data space, opens up: replay, audit, GC, point-in-time reads, billing-by-mutation, federation between ms3t instances.
  • Per-bucket MST history could be made queryable without walking the whole content space.
  • Alternatively: continue inlining structural blocks with body data and add a separate index over mutation events instead of a separate storage space.
  • See open question S3 on Forge #6 in pkg/ms3t/architectural.md ("MST for buckets, registry for buckets") — discussed and deferred during PR #34 because the registry needs SQL-style transactional CAS that Forge doesn't currently provide.

Acceptance

  • A written proposal as markdown enumerating options, trade-offs (storage cost, retrieval cost, consistency model), and a recommendation
  • Recommendation linked from the epic and reviewed before any code lands against it
  • Either the proposal closes this question, or it spawns a follow-up implementation issue

Activity

  1. alanshaw commented on Jun 24, 2026

    @alanshaw
    MemberAuthor

    Decided to store in bucket.

  2. alanshaw commented on Jun 24, 2026

    @alanshaw
    MemberAuthor

    Incoming proposals for

    1. Bucket versioning
    2. Sharding
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions