Repository navigation
fix(iceberg): replace MinIO and the Spark build in both iceberg labs - #307
Open
JakeSCahill wants to merge 2 commits into
Open
JakeSCahill wants to merge 2 commits into
JakeSCahill wants to merge 2 commits into
Conversation
… lab MinIO's images no longer pull from Docker Hub or quay.io, and the custom Spark image no longer builds (bullseye security packages 404, Spark 3.5.7 left the Apache CDN). Use RustFS for object storage and the prebuilt Jupyter Docker Stacks PySpark image, which fetches the Iceberg jars at session start, so there is nothing to build. Redpanda now waits for the bucket, since it aborts startup when the bucket is missing.
Same causes as the Docker Compose lab. A plain RustFS Deployment replaces the MinIO Operator and tenant, and Spark runs from the prebuilt Jupyter image with its config and notebook in a ConfigMap, replacing the build and kind load steps.
✅ Deploy Preview for redpanda-labs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both iceberg labs are broken for everyone right now:
This replaces MinIO with RustFS 1.0.0 (Apache-2.0, S3-compatible, console on port 9001). It also replaces the custom Spark build with the prebuilt
quay.io/jupyter/pyspark-notebook:spark-4.1.2image, which fetches the Iceberg 1.11.0 jars when the first Spark session starts. Users no longer build anything. The old image also bundled taxi data and Scala and Java kernels that nothing used.mcis replaced by anaws-clijob that keeps the bucket anonymously readable. Redpanda now waits for the bucket, because it aborts startup when the bucket is missing. The notebook's%%sqlcells are nowspark.sql(...), so it needs no extra packages.kind loadand verify steps.Tested end to end:
spark-sqland from the notebook run headlessly. The bucket allows anonymous reads and the console returns 200.key_valuetable queries fromspark-sqland the notebook. The RustFS console returns 200 on the NodePort.