Repository navigation
Install the exact shrinkwrapped dependency tree in v5.2 Docker images - #1183
Merged
Merged
Conversation
`npm install --global <tgz>` ignores the npm-shrinkwrap.json inside a local tarball and re-resolves every range, so the images shipped different dependency versions than npm-registry installs. The run stage of Dockerfile, Dockerfile-gpu and Dockerfile-openshift now extracts the tarball into the same global-prefix location, runs `npm ci --omit=dev` there, links the bins with `npm rebuild --global`, and fails the build if the installed tree differs from the shrinkwrap. Dispatch-Task: harper-pro-image-honour-shrinkwrap-52-51 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: harper-pro-image-honour-shrinkwrap-52-51 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Code Review
This pull request introduces a custom installation script and a shrinkwrap validation tool (check-installed-shrinkwrap.mjs) to ensure that installing the packed harper-pro tarball strictly adheres to the dependency tree defined in npm-shrinkwrap.json. The Dockerfiles have been updated to use this new script, and unit tests have been added to verify the validation logic. The review feedback suggests several robustness improvements: skipping hidden files inside scoped directories to avoid spurious failures, gracefully handling missing or malformed package.json files during dependency collection, and guarding the test cleanup hook against uninitialized variables.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
⊙ Problem
The v5.2 Docker images do not contain the dependency versions that npm users get. The release tarball ships an
npm-shrinkwrap.json, but the images install it withnpm install --global <tgz>, and npm ignores a shrinkwrap inside a local tarball. Every image build re-resolves every semver range. A 5.2.15 build drifts on 255 packages (for example@aws-sdk/client-s33.1159.0 instead of 3.1106.0). On v5.1 the same drift shipped msgpackr 2.1.2 in 5.1.30, which lost file-backed Blob data in the image only.🤔 Is a v5.2-only image fix the right scope?
Yes, as specified. main and v5.3 already fixed this by publishing a bundled dependency tree (#981). Backporting that changes the package format on a patch line and needs a core PR per line. This PR changes only how the image installs the tarball it already builds.
💡 Solution
npm-shrinkwrap.json, at its exact versions.harper/harperdbcommands, native install scripts, the uWS copy and the pointer-compression steps are unchanged.⚖️ Alternatives
@harperfast/harper-pro@<version>from the registry — images build from source before publish; PR and pre-release images have no registry version.npm install --global— the build would fail on nearly every run.🔧 Changes
build-tools/install-packed-harper-pro.sh(new): extracts the tarball into$NPM_CONFIG_PREFIX/lib/node_modules/@harperfast/harper-pro, runsnpm ci --omit=devthere, links the bins withnpm rebuild --global, then runs the drift check.npm cirefuses the tarball's package.json because its devDependencies are pruned from the shrinkwrap, so they are dropped for the install and the published package.json is put back afterwards.build-tools/check-installed-shrinkwrap.mjs(new): walks the installednode_modules(scoped and nested) and fails on an extra package, a version mismatch, a missing required package, a symlink, or an unsupported lock / unreadable manifest.Dockerfile(copy, install),Dockerfile-gpu(copy, install),Dockerfile-openshift(copy, install): copy both scripts into the run stage and call the install script in place ofnpm install --global.🤔 Should a missing optional package fail the build?
No, for now. Optional entries may be absent and are listed in the build log.
segfault-handlercannot compile on the GPU and OpenShift bases (no compiler), so a strict rule breaks those builds;npm install --globalalready tolerated this. Tightening to platform-only skips is a one-line change plus an allowlist.🤔 Should the check also run after the uWS copy?
No. uWS is an optional peer outside the shrinkwrap and is added to official images on purpose, so the check runs before that copy and the pointer-compression binary swaps.
✅ Verification
docker buildof all three Dockerfiles, plus the pointer-compression variant (RUN_IMAGE=harperfast/node-pointer-compression:24), on linux/amd64: each passes the drift check (777 packages; 774 on GPU/OpenShift, wheresegfault-handlercannot build) and prints5.2.15fromharper version. arm64 was not built.npm install --globalstep fails the build with 255 problems (175 version, 39 extra, 41 missing).FileBackedBlobwith matching sha256. OpenShift ran as UID 1000740000, GID 0. The smoke fails on an image with msgpackr 2.1.2 swapped in (the record reads back as a placeholder object).checkInstalledShrinkwrap.test.mjs: 10 cases (match, drift at direct/scoped/nested/optional paths, extra, missing, symlink, malformed manifest, unsupported lock).Drift check output and Blob smoke script
Old
npm install --globalstep (5.2.15 build), first lines:New install step:
777 installed packages match npm-shrinkwrap.json.Blob smoke: a component with
type BlobSmokeRec @table { id: ID! @primaryKey payload: Blob }and a resource that writescreateBlob(randomBytes(size))on POST and returns{isBlob, type, size, sha256}of the stored payload on GET. The script mounts it intocomponents/, POSTs, runsdocker stop+docker rm, starts a new container on the same volume, GETs, and compares size and sha256.With msgpackr 2.1.2 swapped into the image:
No milestone: a v5.2 milestone triggers
cherry-pick-patch.ymlonto v5.3, which already has #981 and would conflict. The v5.1 change is #1184.Refs HarperFast/harper#2942
— Claude Opus 5.5
🤖 Generated with Claude Code
Release target: v5.2 — fixes the v5.2 image build only; main/v5.3 already fixed by #981
Related PRs: #1184 overlaps (same change on v5.1)
Complexity: medium
Origin — the dispatch brief this PR was written from
Make the harper-pro Docker images built from the v5.2 and v5.1 branches install exactly the dependency tree recorded in the release tarball's npm-shrinkwrap.json, instead of re-resolving semver ranges at image build time. Open two draft PRs: one based on
v5.2, one based onv5.1. Do NOT target main or v5.3 (already fixed, see context).Acceptance
For each of v5.2 and v5.1: (1)
docker buildofDockerfilesucceeds andharper versionruns in the image. (2) A check, run in the image build so drift fails the build, that every package under the installed harper-pronode_modulesmatches the version in the tarball's npm-shrinkwrap.json (name+version; extra/missing packages fail). Show in the PR body that this check FAILS against the oldnpm install --global <tgz>step (e.g. msgpackr resolves 2.1.x vs shrinkwrap 2.0.x) and passes with the fix. (3)Dockerfile-gpuandDockerfile-openshiftget the same change. (4) The image boots Harper and a Blob round-trips (store a file-backed Blob, restart, read it back as a Blob with the right byte count); a short smoke script is fine. Do not touch main or v5.3, do not change build-pro.sh's shrinkwrap generation, do not change package.json dependency ranges.Dispatch: task
harper-pro-image-honour-shrinkwrap-52-51· queued by kris-session · ran by claude/opus/high · worker kzyp-xps-1Review-Coverage: authored=claude; ran=gemini,cursor-composer,codex,cursor-grok; late=cursor-grok:adjudicated; adjudicated=domain; declined=muse; rounds=2; full=1 @ 1b44623
Review-Attention: read ~3m (decisions: install-mechanism, optional-absence-semantics, check-placement, release-line-scope, do-less-alternative) @ 1b44623