qperf

QEMU plugin and analyzer for guest-kernel performance profiling.

中文文档 · Based on QEMU TCG Plugins

Experimental

What it provides

• Guest-kernel stack sampling in wall-clock or instruction-count mode.
• Optional PulseOS task, scheduler, block/wake, enqueue, and phase tracing.
• Folded-stack, per-vCPU/task, baseline-diff, speedscope, and event-derived off-CPU output.

Requirements

• QEMU 9.2.0 or later (Plugin API v4).
• DWARF debugging information and frame pointers.
• Kernel code and stack addresses in the upper half of the address space by default. Use the kernel plugin argument for another layout.

Build the kernel with debug information and frame pointers:

• Rust: pass -C force-frame-pointers -C debuginfo=2 -C strip=none through RUSTFLAGS or build.rustflags. See the rustc codegen options.
• C: pass -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g to gcc, usually through CFLAGS.

Quick start

1. Build qperf

cargo build --release
cargo install --path analyzer

The plugin is target/release/libqperf.so; the analyzer is qperf-analyzer.

2. Capture a profile

qemu-system-xxx ... -plugin target/release/libqperf.so

The default wall-clock mode samples each vCPU at 99 Hz and writes qperf.bin. Other useful modes:

# Wall-clock sampling; check the deadline once per translated block.
qemu-system-xxx ... -plugin target/release/libqperf.so,freq=101,out=kernel.bin

# Sample every fixed number of guest kernel instructions.
qemu-system-xxx ... -plugin target/release/libqperf.so,mode=icount,period=100000

Each sample records wall-clock nanoseconds and the per-vCPU guest kernel instruction count. mode=icount changes when samples are taken; it does not remove either time domain from the record.

The frame-pointer walker defaults to the inclusive kernel range 0x8000000000000000:0xffffffffffffffff, 128 frames, and a 1 MiB stack span. Adjust these limits when needed:

qemu-system-xxx ... -plugin target/release/libqperf.so,kernel=0xffffffc000000000:0xffffffffffffffff,max_frames=64,max_stack_bytes=65536

Captures use the versioned, checksummed v5 format. Sampling callbacks do not wait for disk I/O. The default bounded queues are 4096 samples (queue=N) and 16384 trace events (event_queue=N); QEMU writes separate sample/event drop statistics on exit.

3. Capture PulseOS task and off-CPU data

Build the trace-enabled artifacts:

make qperf-test

This produces kernel-rv-qperf, kernel-la-qperf, and matching PulseOS_*-qperf.elf files. Ordinary make test artifacts do not enable qperf-trace.

Resolve the trace marker from the same ELF used to build the QEMU kernel:

RV_TRACE=0x$(nm -n PulseOS_riscv64-qemu-virt-qperf.elf | awk '$3 == "__pulse_qperf_trace_v1" { print $1 }')
qemu-system-riscv64 ... \
  -kernel kernel-rv-qperf \
  -plugin target/release/libqperf.so,trace=$RV_TRACE,out=qperf.bin

With trace=ADDRESS, the capture adds task identity, scheduler switches, block/wake/exit events, enqueue placement, phase markers, block-site stacks, and QEMU vCPU idle/resume events. BUILDSTORM_BEGIN/BUILDSTORM_COMPILE select the buildstorm phase; other workloads can emit QPERF_PHASE_BEGIN <name> and QPERF_PHASE_END <name>. Phase names may contain up to 16 ASCII alphanumeric characters, _, ., or -.

Trace callbacks perturb scheduler and blocking paths. Use trace mode for attribution, and compare timings only between runs with the same trace feature, plugin arguments, QEMU topology, image, and workload phase. Without trace, qperf is a sample-only profiler and task identity is unavailable.

Analyze a capture

Basic folded output:

qperf-analyzer -e path/to/kernel.elf path/to/qperf.bin path/to/result.folded

Filter samples and keep per-CPU output:

qperf-analyzer -e kernel.elf \
  --cpu 0,2 --from 1000000 --until 5000000 \
  --group-by-cpu --split-by-cpu \
  qperf.bin result.folded

--from and --until use wall-clock nanoseconds or guest instruction counts according to the capture mode. --baseline old.bin writes differential folded counts; --format speedscope writes speedscope JSON.

For a PulseOS trace capture, add task roots and write an event-derived off-CPU profile. Use --phase-occurrence N when a phase repeats:

qperf-analyzer -e PulseOS_riscv64-qemu-virt-qperf.elf \
  --phase buildstorm --phase-occurrence 1 \
  --group-by-task \
  --off-cpu-output buildstorm.offcpu.folded \
  qperf.bin buildstorm.oncpu.folded

Off-CPU weights are wall-clock nanoseconds. off-cpu/blocked covers blocked switch-out to matching wake; off-cpu/runnable covers wake/preemption to the next run. Enqueue frames include the insertion CPU, target CPU, run-queue depth, and enqueue reason. An unfinished phase is closed at the last captured timestamp and reported as complete=false.

Visualize

• Generate flame graphs with flamegraph.pl or inferno-flamegraph.
• Use speedscope for interactive local viewing.
• Convert folded stacks to pprof with pprofutils folded, then use pprof.me. pprof.me has a 2 MB upload limit; speedscope processes files locally.

Starry OS

• The default BACKTRACE=y build options should enable the debugging support qperf needs.
• Enable the plugin with make ... run QEMU_ARGS="-plugin libqperf.so".

Important limitations

• A nonzero sample-drop count weakens on-CPU data; a nonzero event-drop count makes off-CPU attribution incomplete.
• Folded stacks are inclusive guest-kernel samples, not additive wall-clock time or direct user-mode compiler time.
• Match the kernel, ELF, plugin, analyzer, image, QEMU topology, and workload phase before comparing captures.