Skip to content

ENH: Add benchmark suite#92

Merged
oyamad merged 1 commit into
mainfrom
add-benchmark-suite
Jul 2, 2026
Merged

ENH: Add benchmark suite#92
oyamad merged 1 commit into
mainfrom
add-benchmark-suite

Conversation

@oyamad

@oyamad oyamad commented Jul 2, 2026

Copy link
Copy Markdown
Member

Summary

Add a benchmark suite for the package, so that performance-related PRs can
report standardized before/after comparisons.

  • benchmark/benchmarks.jl: defines SUITE in the standard BenchmarkTools.jl
    format, compatible with PkgBenchmark.jl (and AirspeedVelocity.jl, if we later
    add CI benchmarking)
  • benchmark/Project.toml: dedicated benchmark environment
  • benchmark/README.md: usage instructions (standalone and via PkgBenchmark),
    modeled on GameTheory.jl's benchmark/README.md

What is benchmarked

Three model cases — 1-D stochastic optimal growth with Chebyshev and cubic
spline bases, and the 2-D Santos (1999) model from test/test_cdp_multidim.jl
with a quadratic spline basis — each with seven benchmarks:

Key Description
s_wise_max_one_state Per-state maximization kernel _s_wise_max!
bellman_operator One application of bellman_operator!
compute_greedy One application of compute_greedy!
evaluate_policy One application of evaluate_policy!
set_eval_nodes Evaluation on a non-interpolation grid
solve_PFI End-to-end solve with PFI
solve_VFI_50iter solve with VFI, capped at 50 iterations

Kernel benchmarks use basis coefficients from the converged PFI solution so
inputs are realistic; VFI is capped at 50 iterations so the benchmark measures
a fixed amount of work independently of convergence behavior.

Baseline (Julia 1.12.6, Apple M4 Pro)

Selected results from a verified PkgBenchmark.benchmarkpkg run, to indicate
current allocation behavior:

ID time memory allocations
["1d_cheb", "s_wise_max_one_state"] 22.8 μs 135.6 KiB 1,102
["1d_cheb", "bellman_operator"] 778.7 μs 4.4 MiB 36,452
["1d_cheb", "solve_VFI_50iter"] 182.6 ms 700.3 MiB 5,826,322
["2d_spline", "evaluate_policy"] 1.2 ms 10.3 MiB 112,496
["2d_spline", "solve_VFI_50iter"] 182.3 ms 462.0 MiB 9,305,191

Usage

using PkgBenchmark
jud = judge("ContinuousDPs", "<target>", "<baseline>")
export_markdown("judgement.md", jud)

or standalone: julia --project=benchmark benchmark/benchmarks.jl
(see benchmark/README.md for details).

Verification

  • Full suite runs cleanly via benchmarkpkg("ContinuousDPs") (~1.5 min)
  • Standalone execution verified with --project=benchmark

🤖 Generated with Claude Code (Claude Fable 5)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a dedicated benchmarking setup for ContinuousDPs.jl so performance-focused changes can be measured and compared consistently (standalone runs or via PkgBenchmark).

Changes:

  • Added a BenchmarkTools.jl-style SUITE covering kernel-level operations and end-to-end solve runs for representative 1D/2D models.
  • Introduced a dedicated benchmark environment (benchmark/Project.toml) to keep benchmark dependencies isolated.
  • Documented how to run and compare benchmarks (standalone and via PkgBenchmark) in benchmark/README.md.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
benchmark/benchmarks.jl Defines the benchmark SUITE with model cases and per-kernel / end-to-end benchmarks, plus standalone execution support.
benchmark/Project.toml Adds a dedicated benchmark environment with the required dependencies.
benchmark/README.md Provides usage instructions for running the suite and comparing commits with PkgBenchmark.

@oyamad oyamad merged commit ad6bcb0 into main Jul 2, 2026
6 checks passed
@oyamad oyamad deleted the add-benchmark-suite branch July 2, 2026 08:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants