-
Notifications
You must be signed in to change notification settings - Fork 187
feat(kernel): add build-time lean/net firmware selection #596
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
4764257
5ce56ee
dad37bf
5deede3
323dd9f
a62b3b1
9e5b915
049ff38
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -6,7 +6,8 @@ _ensure-python-deps: | |
| echo "📦 Creating virtual environment..."; \ | ||
| python3 -m venv .venv || { echo "❌ Failed to create virtual environment"; exit 1; }; \ | ||
| fi | ||
| @. .venv/bin/activate && pip install -q uv && (cd sdks/python && uv pip install --group dev --group sync) | ||
| @. .venv/bin/activate && { command -v uv >/dev/null || pip install -q uv; } && \ | ||
| (cd sdks/python && uv pip install --group dev --group sync) | ||
|
Comment on lines
+9
to
+10
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -eu
. .venv/bin/activate
echo "uv: $(command -v uv || true)"
uv --version
uv pip install --help | grep -F -- '--group'
echo "maturin: $(command -v maturin || true)"
maturin --version
maturin develop --help | grep -F -- '--uv'Repository: boxlite-ai/boxlite Length of output: 216 🏁 Script executed: #!/bin/bash
set -u
printf '--- make/dev.mk context ---\n'
if [ -f make/dev.mk ]; then
cat -n make/dev.mk | sed -n '1,60p'
else
echo 'make/dev.mk not found'
fi
printf '\n--- repository files matching activators or tool installs ---\n'
git ls-files | rg '(^|/)(Makefile|dev\.mk|pyproject\.toml|uv\.lock|Cargo\.toml|\.python-version|requirements|Pipfile|poetry\.lock|taskfile|devcontainer|Dockerfile)' || true
printf '\n--- usages of command -v uv/maturin in tracked files ---\n'
rg -n '--command -v (uv|maturin)|command -v (uv|maturin)|VIRTUAL_ENV|/bin/(uv|maturin)' . || trueRepository: boxlite-ai/boxlite Length of output: 4195 🌐 Web query:
💡 Result: Support for the [dependency-groups] table (PEP 735) and the associated --group syntax was introduced in uv version 0.4.27, released on October 25, 2024 [1][2][3]. While the --group flag was initially introduced in 0.4.27 as part of the core project management functionality (e.g., for use with uv add, uv sync, etc.) [2], support for the --group flag within the uv pip install and uv pip compile commands was added later, in uv version 0.6.0 (via PR Citations:
🌐 Web query:
💡 Result: The --uv option in maturin develop was introduced in version 1.6.0 [1][2]. It allows the maturin develop command to use uv (specifically the uv pip install interface) for package installation instead of the default pip [3][4][2]. Key details regarding this option include: - Purpose: It provides a faster, modern alternative to pip for installing your project during development [1][2]. - Mechanism: Maturin does not perform dependency resolution itself; when --uv is used, it builds the Rust extension and then executes uv pip install with the relevant requirements [5]. - Version Requirement: Users must be on maturin version 1.6.0 or higher to access this flag [2]. - Configuration: While you can pass the flag via the command line (maturin develop --uv) [3][4], some users integrate it into their development workflows by using it alongside uv-managed virtual environments or configuration files [5][6]. If you encounter unexpected argument errors, ensure your maturin version is at least 1.6.0 [2]. If you are already on a supported version but still face issues, confirm that uv is available in your system's PATH, as maturin calls the uv CLI as a subprocess [1][2]. Citations:
Ensure the tools come from the virtual environment.
📍 Affects 1 file
🤖 Prompt for AI AgentsSource: MCP tools |
||
|
|
||
| # Ensure Node SDK dependencies are installed (lightweight, no build). | ||
| _ensure-node-deps: | ||
|
|
@@ -31,7 +32,8 @@ _ensure-apps-deps: | |
| # Build wheel locally with maturin + embedded runtime | ||
| dev\:python: $(if $(SETUP_DONE),,runtime\:debug) _ensure-python-deps | ||
| @echo "🔨 Building wheel with maturin (embedded-runtime)..." | ||
| @. .venv/bin/activate && pip install -q maturin && cd sdks/python && maturin develop --uv | ||
| @. .venv/bin/activate && { command -v maturin >/dev/null || uv pip install maturin; } && \ | ||
| cd sdks/python && maturin develop --uv | ||
|
|
||
| dev\:c: $(if $(SETUP_DONE),,runtime\:debug) | ||
| @echo "🔨 Building C SDK (debug)..." | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| #!/usr/bin/env bash | ||
| # Build the built-in "fat" libkrunfw variant required by | ||
| # `boxlite run --kernel-variant net`. | ||
| # | ||
| # Thin wrapper over build-libkrunfw.sh: it pins the net overlay, a distinct | ||
| # SONAME (so the net blob can sit next to the lean one in the embedded runtime | ||
| # without a dlopen identity collision), and the canonical output path that | ||
| # `libkrun-sys/build.rs` auto-detects and embeds on the next `make cli`. | ||
| # | ||
| # Why not download a prebuilt: the net kernel adds ~2 MB of network subsystems | ||
| # on top of the lean kernel. Until upstream boxlite-ai/libkrunfw publishes the | ||
| # net variant in its releases, anyone iterating on `--kernel-variant net` builds it | ||
| # locally. (Set BOXLITE_LIBKRUNFW_NET_PATH only when the blob lives outside the | ||
| # workspace — e.g. CI cache, distro packaging.) | ||
| # | ||
| set -euo pipefail | ||
|
|
||
| SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
| REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" | ||
|
|
||
| ARCH="${ARCH:-$(uname -m)}" | ||
| case "$ARCH" in | ||
| x86_64) OVERLAY_NAME="overlay-net_x86_64" ;; | ||
| aarch64|arm64) OVERLAY_NAME="overlay-net_aarch64" ;; | ||
| *) echo "❌ unsupported ARCH=$ARCH (only x86_64 and aarch64 have a net overlay yet)" >&2; exit 1 ;; | ||
| esac | ||
|
|
||
| OVERLAY="$REPO_ROOT/src/deps/libkrun-sys/net-configs/$OVERLAY_NAME" \ | ||
| SONAME="libkrunfw-net.so.5" \ | ||
| OUT="$REPO_ROOT/target/net-kernel/lib64/libkrunfw-net.so.5" \ | ||
| HINT="Rebuild boxlite to embed it (libkrun-sys/build.rs auto-detects this path): | ||
|
|
||
| make cli | ||
|
|
||
| Then \`boxlite run --kernel-variant net\` loads this kernel instead of the lean one." \ | ||
| ARCH="$ARCH" \ | ||
| exec bash "$SCRIPT_DIR/build-libkrunfw.sh" |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,119 @@ | ||
| #!/usr/bin/env bash | ||
| # Build a libkrunfw kernel blob from a config overlay. | ||
| # | ||
| # This is the implementation behind `make libkrunfw-net`. It merges the net | ||
| # config overlay on top of upstream's lean config, builds the kernel, stamps a | ||
| # distinct SONAME, and stages the resulting `.so` for runtime embedding. | ||
| # | ||
| # Parameters (all via env): | ||
| # OVERLAY path to a config overlay to append on top of the lean config. | ||
| # Required — it's what makes the kernel non-lean. | ||
| # KCONFIG base libkrunfw config NAME under vendor/libkrunfw/ | ||
| # (default: arch lean `config-libkrunfw_<arch>`). | ||
| # SONAME ELF SONAME stamped on the output. | ||
| # OUT output blob path. | ||
| # HINT optional one-line "next step" message printed after the build. | ||
| # DRY_RUN if set, resolve + validate + merge the config and print the | ||
| # plan, but skip the (~10-20 min) kernel build and staging. | ||
| # ARCH override target arch (default: uname -m). | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
| REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" | ||
|
|
||
| LIBKRUNFW_SRC="$REPO_ROOT/src/deps/libkrun-sys/vendor/libkrunfw" | ||
|
|
||
| ARCH="${ARCH:-$(uname -m)}" | ||
| case "$ARCH" in | ||
| x86_64) DEFAULT_KCONFIG="config-libkrunfw_x86_64" ;; | ||
| aarch64|arm64) DEFAULT_KCONFIG="config-libkrunfw_aarch64" ;; | ||
| *) echo "❌ unsupported ARCH=$ARCH" >&2; exit 1 ;; | ||
| esac | ||
|
|
||
| KCONFIG_NAME="${KCONFIG:-$DEFAULT_KCONFIG}" | ||
| SONAME="${SONAME:-libkrunfw.so.5}" | ||
| OUT="${OUT:-$REPO_ROOT/target/net-kernel/lib64/libkrunfw-net.so.5}" | ||
| OVERLAY="${OVERLAY:-}" | ||
|
|
||
| # ── Sanity ────────────────────────────────────────────────────────────────── | ||
|
|
||
| if [ -z "$OVERLAY" ]; then | ||
| echo "❌ OVERLAY is required (path to a config overlay to append)." >&2 | ||
| echo " Set OVERLAY to a file of CONFIG_*=y lines." >&2 | ||
| exit 1 | ||
| fi | ||
| if [ ! -f "$OVERLAY" ]; then | ||
| echo "❌ overlay not found: $OVERLAY" >&2; exit 1 | ||
| fi | ||
|
|
||
| if [ ! -f "$LIBKRUNFW_SRC/Makefile" ]; then | ||
| echo "❌ libkrunfw submodule not initialised at $LIBKRUNFW_SRC" >&2 | ||
| echo " Run: git submodule update --init --recursive src/deps/libkrun-sys/vendor/libkrunfw" >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| LEAN_CONFIG="$LIBKRUNFW_SRC/$KCONFIG_NAME" | ||
| if [ ! -f "$LEAN_CONFIG" ]; then | ||
| echo "❌ base config not found: $LEAN_CONFIG" >&2; exit 1 | ||
| fi | ||
|
|
||
| # ── Merge overlay onto the lean config ────────────────────────────────────── | ||
| # | ||
| # Append the overlay to the lean config. The overlay's `CONFIG_X=y` lines | ||
| # OVERRIDE the lean config's `# CONFIG_X is not set` lines because Kconfig | ||
| # parses sequentially. `make olddefconfig` (run by libkrunfw's Makefile) | ||
| # fills in any dependent options the newly-enabled parents require. | ||
| MERGED_CONFIG="$(mktemp)" | ||
| trap 'rm -f "$MERGED_CONFIG"' EXIT | ||
| cat "$LEAN_CONFIG" "$OVERLAY" > "$MERGED_CONFIG" | ||
|
|
||
| echo "🔧 libkrunfw build plan ($ARCH)" | ||
| echo " base config: $LEAN_CONFIG" | ||
| echo " overlay: $OVERLAY" | ||
| echo " merged size: $(wc -l < "$MERGED_CONFIG") lines" | ||
| echo " soname: $SONAME" | ||
| echo " output: $OUT" | ||
|
|
||
| if [ -n "${DRY_RUN:-}" ]; then | ||
| echo "🟡 DRY_RUN set — validated config merge, skipping kernel build + staging." | ||
| exit 0 | ||
| fi | ||
|
|
||
| # Swap the merged config into the submodule's config in place so libkrunfw's | ||
| # Makefile (which always reads $KCONFIG_NAME) picks it up. Restore on exit so | ||
| # the lean build path isn't permanently polluted with the overlay's lines. | ||
| LEAN_BACKUP="$(mktemp)" | ||
| trap 'cp -f "$LEAN_BACKUP" "$LEAN_CONFIG" 2>/dev/null || true; rm -f "$MERGED_CONFIG" "$LEAN_BACKUP"' EXIT | ||
| cp "$LEAN_CONFIG" "$LEAN_BACKUP" | ||
| cp "$MERGED_CONFIG" "$LEAN_CONFIG" | ||
|
|
||
| # ── Build ─────────────────────────────────────────────────────────────────── | ||
|
|
||
| echo "🔨 Building libkrunfw (this downloads kernel source on first run, ~10-20 min)..." | ||
| cd "$LIBKRUNFW_SRC" | ||
| make -j"$(nproc)" MAKEFLAGS="" | ||
|
|
||
| # ── Stage the result ──────────────────────────────────────────────────────── | ||
|
|
||
| OUT_DIR="$(dirname "$OUT")" | ||
| mkdir -p "$OUT_DIR" | ||
|
|
||
| # libkrunfw's Makefile produces libkrunfw.so.5.<minor>.<patch> with a symlink | ||
| # chain libkrunfw.so.5 → it. Copy the real file and stamp the requested SONAME. | ||
| REAL_BLOB=$(ls "$LIBKRUNFW_SRC"/libkrunfw.so.5.* 2>/dev/null | head -1 || true) | ||
| if [ -z "$REAL_BLOB" ]; then | ||
| echo "❌ build succeeded but couldn't find libkrunfw.so.5.* in $LIBKRUNFW_SRC" >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| cp "$REAL_BLOB" "$OUT" | ||
|
Comment on lines
+102
to
+110
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win Copy the artifact selected by the SONAME symlink. Line 104 picks the lexicographically first historical blob. If multiple versions remain, this can embed an old kernel despite a successful build. Copy Proposed fix-REAL_BLOB=$(ls "$LIBKRUNFW_SRC"/libkrunfw.so.5.* 2>/dev/null | head -1 || true)
+REAL_BLOB="$LIBKRUNFW_SRC/libkrunfw.so.5"
if [ -z "$REAL_BLOB" ]; then
echo "❌ build succeeded but couldn't find libkrunfw.so.5.* in $LIBKRUNFW_SRC" >&2
exit 1
fi
-cp "$REAL_BLOB" "$OUT"
+cp -L "$REAL_BLOB" "$OUT"🧰 Tools🪛 Shellcheck (0.11.0)[info] 104-104: Use find instead of ls to better handle non-alphanumeric filenames. (SC2012) 🤖 Prompt for AI Agents |
||
| patchelf --set-soname "$SONAME" "$OUT" | ||
|
|
||
| echo "" | ||
| echo "✅ Built libkrunfw: $OUT" | ||
| echo " Size: $(du -h "$OUT" | cut -f1) (vs lean: $(du -h "$REAL_BLOB" 2>/dev/null | cut -f1 || echo '?'))" | ||
| if [ -n "${HINT:-}" ]; then | ||
| echo "" | ||
| echo "$HINT" | ||
| fi | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -142,6 +142,51 @@ pub fn configure_library_env(cmd: &mut Command, addr: *const libc::c_void) { | |
| } | ||
| } | ||
|
|
||
| /// Like [`configure_library_env`] but with caller-supplied directories | ||
| /// PRE-pended to the loader search path. Used by `--kernel-variant net` to | ||
| /// inject a per-box dir whose `libkrunfw.so.5` symlinks to the net blob. | ||
| pub fn configure_library_env_with_prepend( | ||
| cmd: &mut Command, | ||
| addr: *const libc::c_void, | ||
| prepend: &[PathBuf], | ||
| ) { | ||
| let mut lib_dirs: Vec<PathBuf> = prepend.iter().filter(|p| p.exists()).cloned().collect(); | ||
|
|
||
| if let Some(runner_dir) = LibraryLoadPath::get(Some(addr)) | ||
| && let Some(dylibs) = runner_dir.parent() | ||
| && dylibs.exists() | ||
| { | ||
| lib_dirs.push(dylibs.to_path_buf()); | ||
| } | ||
|
|
||
| #[cfg(feature = "embedded-runtime")] | ||
| if let Some(runtime) = crate::runtime::embedded::EmbeddedRuntime::get() { | ||
| lib_dirs.push(runtime.dir().to_path_buf()); | ||
| } | ||
|
Comment on lines
+162
to
+165
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win Preserve Unlike 🤖 Prompt for AI Agents |
||
|
|
||
| if lib_dirs.is_empty() { | ||
| return; | ||
| } | ||
|
|
||
| #[cfg(target_os = "macos")] | ||
| { | ||
| let mut paths: Vec<String> = lib_dirs.iter().map(|d| d.display().to_string()).collect(); | ||
| if let Ok(existing) = std::env::var("DYLD_FALLBACK_LIBRARY_PATH") { | ||
| paths.push(existing); | ||
| } | ||
| cmd.env("DYLD_FALLBACK_LIBRARY_PATH", paths.join(":")); | ||
| } | ||
|
|
||
| #[cfg(target_os = "linux")] | ||
| { | ||
| let mut paths: Vec<String> = lib_dirs.iter().map(|d| d.display().to_string()).collect(); | ||
| if let Ok(existing) = std::env::var("LD_LIBRARY_PATH") { | ||
| paths.push(existing); | ||
| } | ||
| cmd.env("LD_LIBRARY_PATH", paths.join(":")); | ||
| } | ||
| } | ||
|
|
||
| pub fn register_to_tracing(non_blocking: NonBlocking, env_filter: EnvFilter) { | ||
| let _ = tracing_subscriber::registry() | ||
| .with(env_filter) | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Align net-kernel documentation with the exposed interfaces. Both locations retain pre-rename commands or variables, so users following the build instructions will use unsupported names.
make/build.mk#L33-L53: document--kernel-variant netandBOXLITE_LIBKRUNFW_NET_PATH.src/deps/libkrun-sys/net-configs/overlay-net_x86_64#L1-L6: replacemake libkrunfw-dindwithmake libkrunfw-net.📍 Affects 2 files
make/build.mk#L33-L53(this comment)src/deps/libkrun-sys/net-configs/overlay-net_x86_64#L1-L6🤖 Prompt for AI Agents