Gate custom kernels behind an RC opt-in

[DorianZheng]

Summary

• move custom-kernel configuration behind the explicit boxlite::experimental namespace
• parse BOXLITE_EXPERIMENTAL=custom-kernel once in the CLI and inject typed feature state into the runtime
• hide custom-kernel flags from help and generated completions while keeping the RC path callable
• enforce the same feature gate for create, start, and restart flows
• document the experimental contract and remove the feature from stable API references

Why

Custom-kernel support is release-candidate functionality. It should remain available for deliberate testing without appearing as a stable, discoverable feature. Unknown experimental tokens fail closed.

Impact

CLI callers must set BOXLITE_EXPERIMENTAL=custom-kernel before using custom-kernel flags. Rust callers opt in through RuntimeBuilder and the experimental API. The REST API does not expose the feature.

Validation

• make clippy
• make fmt:check
• Rust unit suite: 873 BoxLite tests and 47 shared tests passed
• CLI RC coverage: 5 custom-kernel tests and 2 opt-in/token tests passed
• CLI completion coverage: 13 tests passed

The native CLI build was not repeated because this worktree's vendored libkrun submodules are uninitialized; the focused CLI tests used the repository's dependency-stub/linker workaround.

Summary by CodeRabbit

• New Features

  • Added release-candidate support for custom guest kernels, including kernel formats, initramfs, and kernel command-line options.
  • Added explicit opt-in through BOXLITE_EXPERIMENTAL=custom-kernel and Rust SDK configuration.
  • Added validation for unsupported formats, invalid boot artifacts, and unknown experimental feature names.

• Documentation

  • Added experimental custom-kernel usage and compatibility guidance.
  • Updated CLI and API references to reflect its experimental status and limited availability.

• Bug Fixes

  • Persisted custom-kernel configurations now revalidate experimental feature access when restarting or reattaching.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 653bc0c3-90ba-4811-bc10-0e9913a636e6

📥 Commits

Reviewing files that changed from the base of the PR and between 2b717a4 and 663229d.

📒 Files selected for processing (22)

• docs/architecture/README.md
• docs/experimental/custom-kernel.md
• docs/reference/cli/README.md
• docs/reference/rust/README.md
• src/boxlite/src/experimental.rs
• src/boxlite/src/experimental/custom_kernel.rs
• src/boxlite/src/lib.rs
• src/boxlite/src/litebox/init/mod.rs
• src/boxlite/src/litebox/init/tasks/boot_assets.rs
• src/boxlite/src/rest/runtime.rs
• src/boxlite/src/runtime/advanced_options.rs
• src/boxlite/src/runtime/core.rs
• src/boxlite/src/runtime/options.rs
• src/boxlite/src/runtime/rt_impl.rs
• src/boxlite/src/vmm/krun/engine.rs
• src/boxlite/src/vmm/mod.rs
• src/cli/README.md
• src/cli/src/cli.rs
• src/cli/src/commands/create.rs
• src/cli/src/commands/run.rs
• src/cli/src/main.rs
• src/cli/tests/custom_kernel_rc.rs

---

📝 Walkthrough

Walkthrough

This PR moves custom-kernel support into an experimental module, adds explicit BOXLITE_EXPERIMENTAL feature gating, propagates capabilities through local runtime construction and persisted options, hides related CLI flags, and documents the RC behavior.

Changes

Custom kernel RC

Layer / File(s)	Summary
Experimental kernel contract src/boxlite/src/experimental*, src/boxlite/src/lib.rs, src/boxlite/src/runtime/options.rs, src/boxlite/src/runtime/advanced_options.rs	Adds experimental feature parsing and custom-kernel configuration with format detection, validation, architecture-specific constraints, and relocated public types.
Runtime feature propagation and boot integration src/boxlite/src/runtime/*, src/boxlite/src/litebox/init/*, src/boxlite/src/vmm/*, src/boxlite/src/rest/runtime.rs	Injects experimental capabilities into local runtimes, validates persisted and runtime options, and updates boot/VMM consumers to use the experimental kernel types.
CLI opt-in and hidden custom-kernel flags src/cli/src/*, src/cli/tests/custom_kernel_rc.rs	Parses BOXLITE_EXPERIMENTAL, applies selected capabilities during runtime creation, gates custom-kernel options, hides related help/completion entries, and tests missing or unknown feature handling.
Experimental feature documentation docs/architecture/README.md, docs/experimental/*, docs/reference/*, src/cli/README.md	Documents the custom-kernel RC feature and removes it from stable CLI and Rust reference sections.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant RuntimeBuilder
  participant BoxliteRuntime
  participant LocalRuntime
  CLI->>CLI: Parse BOXLITE_EXPERIMENTAL
  CLI->>RuntimeBuilder: Apply ExperimentalFeatures
  RuntimeBuilder->>BoxliteRuntime: Build runtime
  BoxliteRuntime->>LocalRuntime: Create with feature state
  LocalRuntime->>LocalRuntime: Validate custom-kernel options

Loading

Possibly related PRs

• boxlite-ai/boxlite#1041: Refactors and gates the existing custom guest kernel implementation across CLI, runtime, and REST behavior.

Suggested reviewers: g4614

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch agent/support-custom-kernel

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[cla-assistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

tester seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[cla-assistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

tester seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[boxlite-agent]

📦 BoxLite review — looks good · 663229d

Review evidence

• ✅ git diff --numstat origin/main...HEAD && git diff origin/main...HEAD — 22 files changed, reviewed full diff
• ✅ grep -rn stale boxlite::KernelFormat/KernelOptions refs — no leftover references to old public path
• ⚪ cargo test -p boxlite / cargo check — no rust toolchain (cargo/rustc) installed in sandbox

Risk notes

• gate coverage across create/get_or_create/restart — create & get_or_create call sanitize_local_options->require_for_options; restart/start go through BoxBuilder::new->validate_persisted_options using runtime.experimental_features; CLI run/create call KernelFlags::require_enabled pre-flight; consistent
• CLI env parsing timing — main.rs sets cli.global.experimental_features from env after Cli::parse() but before command dispatch (except completion, which doesn't need it); GlobalFlags derives Default via #[arg(skip)] so no uninitialized-state risk
• REST runtime — rest/runtime.rs test still asserts REST rejects kernel config; behavior unchanged, only type path renamed
• format-detection logic — KernelFormat::detect/validate_resolved_format code moved verbatim from options.rs to experimental/custom_kernel.rs, no logic changes observed
• coverage — docs-only diffs (README/architecture/reference) and CLI help/completion-hiding tests sampled, not exhaustively re-verified line by line; no toolchain available so no compile/test run was possible, relied on grep-based cross-reference check

src/boxlite/src/experimental.rs
  ExperimentalFeatures/RuntimeBuilder  +189/-0  new opt-in gating module
src/boxlite/src/experimental/custom_kernel.rs
  KernelOptions/KernelFormat  +331/-0  moved from runtime/options.rs verbatim
src/boxlite/src/runtime/options.rs
  KernelOptions/KernelFormat removal  +2/-314  types relocated out
src/boxlite/src/runtime/rt_impl.rs
  sanitize_local_options/new_with_experimental_features  +46/-3  threads feature gate into create paths
src/boxlite/src/litebox/init/mod.rs
  validate_persisted_options  +35/-1  gates restart/start persisted options
src/cli/src/cli.rs
  KernelFlags::require_enabled/completion_projection  +165/-31  hides RC flags from help/completions, gate check
src/cli/src/main.rs
  experimental_features_from_env wiring  +9/-1  parses BOXLITE_EXPERIMENTAL before dispatch
src/cli/tests/custom_kernel_rc.rs
  integration tests  +42/-0  new CLI-level opt-in tests
docs/*
  varies  docs moved to experimental/custom-kernel.md

_{reviewed 663229d in a BoxLite microVM · @boxlite-agent review to re-run · powered by BoxLite}