Repository navigation
chore: use rustfs for iceberg rest catalog - #1692
Open
TylerHillery wants to merge 1 commit into
Open
TylerHillery wants to merge 1 commit into
TylerHillery wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
The backend swap requires live RustFS 1.0.0 acceptance testing of bucket registration, SigV4 authentication, and create/drop behavior across both shards, but RustFS is unavailable in this review environment.
No open issues 🎉
What changed in this PR
Replaces the standalone local Iceberg REST catalog with RustFS’s built-in catalog, reducing container count and enabling local testing across two table-storage shards.
Changes:
- Configures local and test environments to use RustFS with two warehouse buckets.
- Makes the SigV4 service name and purge-on-drop behavior configurable while preserving AWS defaults.
- Tests the new client options and corrects the acceptance-test partition specification field.
| File | Description |
|---|---|
src/storage/protocols/iceberg/catalog/rest-catalog-client.ts |
Supports configurable signing service and purge behavior. |
src/storage/protocols/iceberg/catalog/rest-catalog-client.test.ts |
Tests signing-service selection and purge query handling. |
src/storage/protocols/iceberg/catalog/index.ts |
Passes the configured service name into SigV4 authentication. |
src/config.ts |
Adds signing-service and purge-on-drop settings. |
docker-compose.yml |
Switches to RustFS and removes the standalone catalog service. |
acceptance/specs/iceberg.test.ts |
Uses partition-spec in table creation payloads. |
.env.test.sample |
Configures tests for RustFS and two warehouse shards. |
.docker/docker-compose-infra.yml |
Initializes both RustFS table buckets and removes the catalog fixture. |
Review effort: Balanced
Contributor
Author
|
@claude always review |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Coverage Report for CI Build 42908125259Coverage decreased (-0.01%) to 87.589%Details
Uncovered ChangesNo uncovered changes found. Coverage Regressions2 previously-covered lines in 1 file lost coverage.
Coverage Stats
💛 - Coveralls |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What kind of change does this PR introduce?
Chore
What is the current behavior?
Currently we use
apache/iceberg-rest-fixture:1.10.1for local iceberg rest catalogWhat is the new behavior?
Swap to rustfs which has built in s3 tables feature and iceberg rest catalog. One less contianer to run for local development and lets us test the s3 table sharding logic.
Additional context
N/A