Save workload images in the container, without SSH or control-plane data transfer

[zoroyihan7]

What

Save Image (OpsJob exportimage) no longer SSHes to the node to run nerdctl commit/login/push, and no image data passes through the API server or the resource manager. The container computes and uploads its own changes; the resource manager only issues a short-lived token, starts one command through pods/exec, and puts the image together in the registry with kilobyte-sized requests. The same path serves a container on a kubelet and one on a virtual kubelet.

Why

• SSH + nerdctl only works on nodes the platform can SSH into and that run containerd, and it copies the registry credential onto the node.
• nerdctl commit captures everything in the container, including what the platform launcher installs at start (openssh-server, socat, curl and dependencies, several thousand files), SSH host keys among them.
• An earlier revision of this PR streamed the layer back over pods/exec and read every base layer from the registry to find deletions. Measured on a real cluster, that cost the API server about 14 CPU-seconds per GB streamed (3–4 cores for the duration) and grew its RSS by ~390 MB, and listing an 11.7 GB base image took 218 s and 266 CPU-seconds in the resource manager, per export.

Data flow

1. At container start the launcher runs /shared-data/save-image record as its first step (before the AINIC/bnxt/authoring bootstrap). It lists the root file system (no other file system, no mount point, no /proc, /sys, /dev) into /shared-data/save-image.base, atomically. /shared-data is a mount, so the record is never part of a saved image. A failure only disables saving for that container.
2. Probe (exec sh -c, bytes): uid, presence of the binary and the record. A container started before this change has neither and is refused with "restart the workload, then save it again". A non-root container is refused.
3. Base: the digest in status.containerStatuses[].imageID must be in the cluster's registry (same repository path); otherwise the job fails rather than copying the base. Only its manifest and config are read.
4. Token: the resource manager asks the registry's token service for repository:<staging project>/<job name>:push,pull, minted with a separate credential limited to the staging project. Harbor gives a bearer token the power of the account that minted it, whatever repository the token names (measured: a token minted with an administrator credential for one repository pushed to another), so the platform's own credential is never used for this, and saving is refused when no staging-only credential is configured. The grant is also read back and refused unless it names the staging repository alone.
5. Export (exec /shared-data/save-image export): the token, the registry CA and the deadline (token expiry minus a minute) go on standard input only, never argv, environment or the Pod spec. Inside the container the binary:
   • lists the file system; paths whose ctime is later than the launcher's .run.sh (written after the launcher's own bootstrap, immediately before the user's entry point) are the changes;
   • paths in the start record and gone now become OCI whiteouts (collapsed under a deleted directory or a directory replaced by a file);
   • leaves out mounts, /proc /sys /dev, /etc/hosts /etc/hostname /etc/resolv.conf, /etc/ssh/ssh_host_*, sockets;
   • reconciles /var/lib/dpkg/status with the files the image holds (drops the launcher's packages);
   • writes one tar layer (numeric owners, hard links, no atime/ctime) and streams it gzip-compressed to the staging repository, TLS only, against the given CA, never retried over plain HTTP;
   • prints the layer digest, diff ID, size and counts.
6. Assembly (resource manager, registry API only): the staging blob is checked to exist with the reported size; the config and manifest are built from the base (pinned by the controller) plus that one layer; the base layers and the new layer are cross-repository mounted into <registry>/<target project>/<ns>/<repo>; a registry that answers a mount with an upload has the upload cancelled and the job fails, so no blob is ever copied through the resource manager. Only the config (KB) and manifest are written. The job's digest output is what the registry reports for the tag, not what the container printed.

What the container can still forge is its own layer's content and diff ID, which only affects the user's own image.

Configuration and deployment prerequisites

New chart values, rendered into the resource manager's and job manager's config:

save_image:
  clusters:
  - cluster: <cluster id>
    registry: <registry host>          # default: the default registry
    target_project: custom             # default
    staging_project: save-staging      # default
    ca_secret: <ns>/<name>             # key ca.crt; default: the built-in Harbor's CA
    staging_push_secret: <ns>/<name>   # default: primus-safe/primus-safe-save-image-staging-cred
    host_aliases:                      # added to every Pod the platform creates in the cluster
    - ip: <registry IP>
      hostnames: [<registry host>]

Before this is deployed, the registry administrator needs to:

• create the staging project (private) in each cluster's registry; a retention rule and periodic garbage collection reclaim staged blobs (once mounted into a saved image they are referenced by it);
• create an account that can push to the staging project only (a Harbor robot account of that project) and store it as a Docker config Secret (config.json or .dockerconfigjson) named by staging_push_secret; never an administrator account;
• make sure the containers of each cluster can reach the registry over TLS: set host_aliases where the cluster's DNS does not resolve the registry name, and ca_secret where its CA differs from the built-in one;
• keep the images workloads start from in that registry (same repository path and digest), or saving them is refused;
• the target project (custom) is still created public by the apiserver when missing, as before.

The preprocess image gains the static save-image binary (a Go build stage); the build workflow now rebuilds it when the binary's sources change. Containers started before the new preprocess image is rolled out cannot be saved until restarted.

Kept from the previous revision

Launcher hand-over boundary (.run.sh ctime), exclusions, dpkg reconciliation, refusal of non-root containers, update permission on the workload, public custom project creation, digest output, unique <YYYYMMDDHHMMSS>-<6 hex> tags.

Removed: reading the base file set from the registry (BaseFileSet), streaming the layer back over exec, GNU find/tar as dependencies, and the pod pull secrets in the keychain (the base must be readable with the platform's credential anyway, to be mounted).

Tests

• Agent: walk/exclusions, baseline record and its atomic replacement, layer contents (hard links, symlinks, whiteouts, headers), refusals (non-root, no record, truncated record, no .run.sh, missing token, wrong CA, plain HTTP), dpkg reconciliation; an end-to-end upload to a TLS registry.
• Controller: a fake Harbor-like registry with a token service, per-repository blobs and cross-repository mounts runs the full export with the agent in-process on a real directory tree, then flattens the published image and checks additions, modifications, deletions, volume content, launcher files, host keys and dpkg; it also counts bytes per token: the layer is uploaded by the container's token while the controller's requests stay under 64 KB. Further tests: old container, non-root, base not in the registry, a token granting more, no staging credential, layer size/digest mismatch, a registry that will not mount (nothing copied, nothing published), the token kept out of errors.
• Config/chart: the values render into both services and parse; the job manager adds host aliases (external/virtual-kubelet Pods included) after the template's own.
• Each guard was mutated and the tests went red, then reverted.
• Real cluster, containerd node, launcher-started root container after an openssh-server/socat/curl bootstrap, with a 1 GiB random file added, a file modified, a file and a directory deleted: export took 10.5–13.8 s for a 1.07 GB layer (six runs); the controller side used ~30 ms CPU, ~29 KB of registry traffic and ~2.4 KB over exec per export; API server CPU during exports stayed within its idle range (no sustained rise, against ~14 CPU-s/GB before). The image ran on another node with the 1 GiB file's checksum intact, the deletions applied, a sibling of the deleted directory kept, and no sshd, host keys or .run.sh. A container without the record was refused.

Not covered

• Virtual-kubelet nodes: the binary runs from /shared-data there and records the file system, but the container could not reach the registry in the test environment (name not resolvable without host aliases); the end-to-end upload there is unverified.
• A staging-only robot account's token was not exercised against a real Harbor (the measurement above used an administrator token in a test container).
• Token lifetime is the registry's (Harbor default 30 minutes); a layer whose upload outlasts it fails.
• The container is not paused while it is read; a file that changes during the read is archived at the size it had when listed.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 71.69363% with 595 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...rce-manager/pkg/ops_job/exportimage/agent/agent.go	75.52%	52 Missing and 29 partials ⚠️
...rce-manager/pkg/ops_job/exportimage/agent/layer.go	58.62%	54 Missing and 18 partials ⚠️
...source-manager/pkg/ops_job/exportimage/assemble.go	60.95%	31 Missing and 26 partials ⚠️
SaFE/resource-manager/cmd/save-image/main.go	0.00%	53 Missing ⚠️
...ce-manager/pkg/ops_job/exportimage/agent/upload.go	73.46%	34 Missing and 18 partials ⚠️
...-manager/pkg/ops_job/exportimage_job_controller.go	79.68%	33 Missing and 19 partials ⚠️
...resource-manager/pkg/ops_job/exportimage/export.go	78.16%	31 Missing and 19 partials ⚠️
...server/pkg/handlers/image-handlers/save_staging.go	69.93%	29 Missing and 20 partials ⚠️
...urce-manager/pkg/ops_job/exportimage/agent/walk.go	66.66%	22 Missing and 10 partials ⚠️
...E/resource-manager/pkg/ops_job/exportimage/exec.go	0.00%	22 Missing ⚠️
... and 9 more

📢 Thoughts on this report? Let us know!