Skip to content

Install the exact shrinkwrapped dependency tree in v5.2 Docker images - #1183

Merged
kriszyp merged 2 commits into
v5.2from
fix/image-honour-shrinkwrap-v5.2
Oct 30, 2026
Merged

kriszyp merged 2 commits into
v5.2from
fix/image-honour-shrinkwrap-v5.2

Conversation

@kriszyp

@kriszyp kriszyp commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

⊙ Problem

The v5.2 Docker images do not contain the dependency versions that npm users get. The release tarball ships an npm-shrinkwrap.json, but the images install it with npm install --global <tgz>, and npm ignores a shrinkwrap inside a local tarball. Every image build re-resolves every semver range. A 5.2.15 build drifts on 255 packages (for example @aws-sdk/client-s3 3.1159.0 instead of 3.1106.0). On v5.1 the same drift shipped msgpackr 2.1.2 in 5.1.30, which lost file-backed Blob data in the image only.

🤔 Is a v5.2-only image fix the right scope?

Yes, as specified. main and v5.3 already fixed this by publishing a bundled dependency tree (#981). Backporting that changes the package format on a patch line and needs a core PR per line. This PR changes only how the image installs the tarball it already builds.

💡 Solution

  • All three Dockerfiles (standard, GPU, OpenShift) now install the tree in the tarball's npm-shrinkwrap.json, at its exact versions.
  • The image build fails on any extra package, version mismatch or missing required package. Optional packages npm skips (wrong platform, failed native build) are logged, not failed.
  • Install location, the harper/harperdb commands, native install scripts, the uWS copy and the pointer-compression steps are unchanged.

⚖️ Alternatives

  • Backport Publish harper-pro with a locked JavaScript dependency bundle instead of a shrinkwrap #981's bundled dependencies — changes the published package format on a patch line; needs a coordinated core PR.
  • Install @harperfast/harper-pro@<version> from the registry — images build from source before publish; PR and pre-release images have no registry version.
  • Pin every dependency in package.json — pins direct dependencies only; most of the drift is transitive.
  • Drift check only, keeping npm install --global — the build would fail on nearly every run.

🔧 Changes

🤔 Should a missing optional package fail the build?

No, for now. Optional entries may be absent and are listed in the build log. segfault-handler cannot compile on the GPU and OpenShift bases (no compiler), so a strict rule breaks those builds; npm install --global already tolerated this. Tightening to platform-only skips is a one-line change plus an allowlist.

🤔 Should the check also run after the uWS copy?

No. uWS is an optional peer outside the shrinkwrap and is added to official images on purpose, so the check runs before that copy and the pointer-compression binary swaps.

✅ Verification

  • docker build of all three Dockerfiles, plus the pointer-compression variant (RUN_IMAGE=harperfast/node-pointer-compression:24), on linux/amd64: each passes the drift check (777 packages; 774 on GPU/OpenShift, where segfault-handler cannot build) and prints 5.2.15 from harper version. arm64 was not built.
  • Negative control: the same check after the old npm install --global step fails the build with 255 problems (175 version, 39 extra, 41 missing).
  • Blob smoke on all four images: a 200 KB file-backed Blob is written, the container is stopped and replaced on the same volume, and the Blob reads back as a FileBackedBlob with matching sha256. OpenShift ran as UID 1000740000, GID 0. The smoke fails on an image with msgpackr 2.1.2 swapped in (the record reads back as a placeholder object).
  • checkInstalledShrinkwrap.test.mjs: 10 cases (match, drift at direct/scoped/nested/optional paths, extra, missing, symlink, malformed manifest, unsupported lock).
  • PR QA: skipped @ 1b44623 — build/packaging only; the image-level checks above are the end-to-end route.
  • Docs: none needed; no user-facing behavior changes.
Drift check output and Blob smoke script

Old npm install --global step (5.2.15 build), first lines:

Installed dependencies differ from npm-shrinkwrap.json in /home/harperdb/.npm-global/lib/node_modules/@harperfast/harper-pro:
  version: node_modules/@aws-sdk/client-s3 is 3.1159.0, shrinkwrap has 3.1106.0
  version: node_modules/fastify is 5.12.7, shrinkwrap has 5.11.3
  version: node_modules/pino-abstract-transport is 1.2.0, shrinkwrap has 3.0.0
  extra: node_modules/fastify/node_modules/pino-abstract-transport@3.0.0
  missing: node_modules/pino/node_modules/pino-abstract-transport@1.2.0

New install step: 777 installed packages match npm-shrinkwrap.json.

Blob smoke: a component with type BlobSmokeRec @table { id: ID! @primaryKey payload: Blob } and a resource that writes createBlob(randomBytes(size)) on POST and returns {isBlob, type, size, sha256} of the stored payload on GET. The script mounts it into components/, POSTs, runs docker stop + docker rm, starts a new container on the same volume, GETs, and compares size and sha256.

written:  {"size":200000,"sha256":"54bf18b5…"}
read back: {"exists":true,"isBlob":true,"type":"FileBackedBlob","size":200000,"sha256":"54bf18b5…"}
PASS: FileBackedBlob of 200000 bytes, sha256 matches

With msgpackr 2.1.2 swapped into the image:

read back: {"exists":true,"isBlob":false,"type":"Object","value":{"description":"Blobs that are not of type text/* can not be directly serialized as JSON, …"},"sha256":null}
FAIL: Blob did not round-trip

No milestone: a v5.2 milestone triggers cherry-pick-patch.yml onto v5.3, which already has #981 and would conflict. The v5.1 change is #1184.

Refs HarperFast/harper#2942

— Claude Opus 5.5

🤖 Generated with Claude Code

Release target: v5.2 — fixes the v5.2 image build only; main/v5.3 already fixed by #981

Related PRs: #1184 overlaps (same change on v5.1)
Complexity: medium

Origin — the dispatch brief this PR was written from

Make the harper-pro Docker images built from the v5.2 and v5.1 branches install exactly the dependency tree recorded in the release tarball's npm-shrinkwrap.json, instead of re-resolving semver ranges at image build time. Open two draft PRs: one based on v5.2, one based on v5.1. Do NOT target main or v5.3 (already fixed, see context).

Acceptance

For each of v5.2 and v5.1: (1) docker build of Dockerfile succeeds and harper version runs in the image. (2) A check, run in the image build so drift fails the build, that every package under the installed harper-pro node_modules matches the version in the tarball's npm-shrinkwrap.json (name+version; extra/missing packages fail). Show in the PR body that this check FAILS against the old npm install --global <tgz> step (e.g. msgpackr resolves 2.1.x vs shrinkwrap 2.0.x) and passes with the fix. (3) Dockerfile-gpu and Dockerfile-openshift get the same change. (4) The image boots Harper and a Blob round-trips (store a file-backed Blob, restart, read it back as a Blob with the right byte count); a short smoke script is fine. Do not touch main or v5.3, do not change build-pro.sh's shrinkwrap generation, do not change package.json dependency ranges.

Dispatch: task harper-pro-image-honour-shrinkwrap-52-51 · queued by kris-session · ran by claude/opus/high · worker kzyp-xps-1

Review-Coverage: authored=claude; ran=gemini,cursor-composer,codex,cursor-grok; late=cursor-grok:adjudicated; adjudicated=domain; declined=muse; rounds=2; full=1 @ 1b44623

Review-Attention: read ~3m (decisions: install-mechanism, optional-absence-semantics, check-placement, release-line-scope, do-less-alternative) @ 1b44623

kriszyp and others added 2 commits October 28, 2026 14:58
`npm install --global <tgz>` ignores the npm-shrinkwrap.json inside a local
tarball and re-resolves every range, so the images shipped different
dependency versions than npm-registry installs. The run stage of Dockerfile,
Dockerfile-gpu and Dockerfile-openshift now extracts the tarball into the
same global-prefix location, runs `npm ci --omit=dev` there, links the bins
with `npm rebuild --global`, and fails the build if the installed tree differs
from the shrinkwrap.

Dispatch-Task: harper-pro-image-honour-shrinkwrap-52-51
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: harper-pro-image-honour-shrinkwrap-52-51
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a custom installation script and a shrinkwrap validation tool (check-installed-shrinkwrap.mjs) to ensure that installing the packed harper-pro tarball strictly adheres to the dependency tree defined in npm-shrinkwrap.json. The Dockerfiles have been updated to use this new script, and unit tests have been added to verify the validation logic. The review feedback suggests several robustness improvements: skipping hidden files inside scoped directories to avoid spurious failures, gracefully handling missing or malformed package.json files during dependency collection, and guarding the test cleanup hook against uninitialized variables.

@kriszyp
kriszyp marked this pull request as ready for review October 30, 2026 13:09
@kriszyp
kriszyp requested review from a team as code owners October 30, 2026 13:09
@kriszyp
kriszyp merged commit d199113 into v5.2 Oct 30, 2026
58 checks passed
@kriszyp
kriszyp deleted the fix/image-honour-shrinkwrap-v5.2 branch October 30, 2026 13:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant