ci: publish nightly SNAPSHOT jars to repository.apache.org - #5902
ci: publish nightly SNAPSHOT jars to repository.apache.org#5902andygrove wants to merge 4 commits into
Conversation
Add a scheduled workflow that builds the Comet native library for linux/amd64 and linux/aarch64 inside an Ubuntu 20.04 container, matching the glibc baseline of the release builder, and deploys SNAPSHOT jars for Spark 3.4, 3.5, 4.0 and 4.1 to the ASF snapshot repository. The run skips when main has not changed since the previous night, and a dry_run dispatch builds and verifies the jars without touching Nexus so the workflow can be exercised on a fork. Document how to consume the snapshots in the installation guide.
github-actions
Bot
added
build
| fi | ||
| for jar in jars/*.jar; do | ||
| for lib in linux/amd64 linux/aarch64; do | ||
| if ! unzip -l "$jar" | grep -q "org/apache/comet/$lib/libcomet.so"; then |
There was a problem hiding this comment.
This can report a missing library when the entry is present. I reran this step against the four jars from your dry run in an Ubuntu 24.04 amd64 container: all eight checks failed with PIPESTATUS=141 0. grep -q exits after its match, leaving unzip to receive SIGPIPE, which pipefail treats as failure. The hosted run passed; this depends on pipe/consumer scheduling.
Letting grep consume the full listing passes all four jars and still rejects a copy with the aarch64 library removed:
| if ! unzip -l "$jar" | grep -q "org/apache/comet/$lib/libcomet.so"; then | |
| if ! unzip -l "$jar" | grep -F "org/apache/comet/$lib/libcomet.so" > /dev/null; then |
| # property overrides every module's setting. $profiles is a | ||
| # space-separated list of -P flags, so it must word-split. | ||
| # shellcheck disable=SC2086 | ||
| JAVA_HOME="$jdk" ./mvnw -B "$GOAL" -DskipTests -Dmaven.deploy.skip=false $profiles < /dev/null |
There was a problem hiding this comment.
Could we validate the packaged jars before deploying them? With GOAL=deploy, this call uploads each snapshot before the native-library check below runs. I confirmed the upload order by deploying the Spark 4.1 reactor to a local file repository.
If that check catches an incomplete jar, the job fails after the jar is already available to consumers. Please move validation ahead of publication so the deployed artifacts have passed the check.
2 participants
Which issue does this PR close?
Closes #5899.
Rationale for this change
Anyone who wants to try a fix or feature before the next release currently has to build Comet from source, including the Rust native library. Publishing nightly SNAPSHOT jars to the ASF snapshot repository gives users and downstream projects a ready-made artifact for the current development version, the same way Spark and Iceberg publish their snapshots.
The repository previously had a tag-triggered GHCR Docker publish workflow, removed in #4241. Its runs had failed for months, at first within a minute of starting and later as
startup_failurefrom unpinned third-party actions, and it built arm64 under QEMU with caching disabled. This workflow avoids all three problems: it only usesactions/*and the repo's own composite actions, builds each architecture on a native runner, and caches the Cargo registry and Maven repository.What changes are included in this PR?
A new scheduled workflow,
.github/workflows/publish_snapshot.yml, with three jobs:changesskips the nightly run whenmainhas had no commits since the previous night, so Nexus does not accumulate identical snapshots. A manual dispatch always runs.nativebuildslibcomet.sofor linux/amd64 onubuntu-24.04and linux/aarch64 onubuntu-24.04-arm. Both build inside anubuntu:20.04container so the library links against glibc 2.31, the same baseline as the release builder indev/release/comet-rm/Dockerfile, and a step fails the job if the library ever requires a newer glibc. The samemake core-*-libstargets as the release are used, so the baseline CPU targets match released jars.deployplaces both libraries underspark/target/classes/org/apache/comet/linux/the waybuild-release-comet.shdoes, then runs./mvnw deployfor the four default variants: Spark 3.4 and 3.5 with Scala 2.12, Spark 4.0 and 4.1 with Scala 2.13. Spark 3.4 is built with JDK 11 and the rest with JDK 17, matchingpr_build_linux.yml.-Dmaven.deploy.skip=falseoverrides the root pom so the parent pom is deployed too, since consumers need it to resolve the child poms and the release publishes it. Before deploying, every jar is checked to contain both native libraries.Credentials come from the
NEXUS_USERandNEXUS_PWrepository secrets that ASF Infra provisions for snapshot publishing, read into asettings.xmlthrough${env.*}rather than written to disk. Theorg.apache:apacheparent pom already maps SNAPSHOT deploys toapache.snapshots.https, so no pom changes are needed. If the secrets are not yet configured on this repository, the deploy step fails at upload and an INFRA ticket is needed.A
dry_runinput onworkflow_dispatchruns the whole pipeline but ends withinstallinstead ofdeployand uploads the jars as workflow artifacts. It is also allowed on forks so the workflow can be exercised before a change lands.The installation guide's snapshot-only section now explains where the snapshots live, which artifacts exist, that they are unreleased builds for testing only, and how to use one with
spark-shelleither by downloading the jar or via--packageswith the snapshot repository. The workflows README lists the new standalone workflow.How are these changes tested?
actionlint,prettier --checkanddev/ci/check-ci-config.pypass locally../mvnw deploy -Dmaven.deploy.skip=false -Pspark-4.1 -DaltDeploymentRepository=...into a file repository confirmed that the parent pom,comet-common,comet-sparkandcomet-spark-integrationare all deployed with timestamped snapshot names.dry_runof the workflow on my fork built both native libraries in the Ubuntu 20.04 containers, passed the glibc check, built all four variants and verified that each jar bundles both libraries: https://github.com/andygrove/datafusion-comet/actions/runs/34770970719Cost per night from that run: each native build takes about 17 minutes on its runner, and the deploy job about 15 minutes for the four Maven builds, so roughly 50 runner-minutes and 32 minutes of wall clock. The Cargo registry and Maven caches were cold on the fork.
The first real publish will be a manual
workflow_dispatchafter this merges, followed by checking that the coordinates resolve from the snapshot repository.