Skip to content

microvm: slim the guest image (kata-agent and rootfs.img) #1899

Description

@ziyue-101

Problem

ateom-microvm boots the guest with kata-agent as PID 1 (init=/usr/bin/kata-agent on Cloud Hypervisor ≥ v53), so the guest uses little of the stock kata guest image:

  • rootfs.img is 256 MiB, mostly Ubuntu userland such as systemd and chrony. The guest only needs kata-agent, the tools DebugConsoleDump runs, iptables, and their shared libraries. Each node still downloads, sha256-verifies and caches the whole image before its first micro-VM actor.
  • kata-agent is the largest binary in the guest. Kata's release build enables the OPA policy engine and initdata, and ateom uses neither. Every cold boot loads the agent's pages, and every memory snapshot carries them.

An earlier experiment rebuilt only the agent without those features (kata 4.0.0, GKE amd64, counter demo, medians of 7–9 cold bakes):

agent binary agent_dial since_boot golden snapshot (compressed)
stock 30.63 MB 377 ms 498 ms 27.7 MiB
rebuilt 18.59 MB 356 ms 468 ms 24.3 MiB

agent_dial and since_boot come from ateom's "Actor boot phases" log line:

  • agent_dial: from the vsock socket appearing until the agent answers;
  • since_boot: from BootVM until the wakeup probe passes.

Proposal

  1. Opt-in build flag. SLIM_ROOTFS=true hack/install-microvm-deps.sh --install builds the slim image locally from the pinned kata release .The flag is off by default, and a path-filtered CI job smoke-tests it.

    • kata-agent is rebuilt without the policy engine and initdata: 30.8 → 15.5 MB;
    • rootfs.img is repacked as a journal-less ext4 image with only what the guest runs: 256 → 32 MiB
  2. Benchmark cold start, suspend and resume on stock vs slim.

  3. Decide from the numbers whether to keep it opt-in, make it the default, or ship prebuilt slim assets.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions