ENH: Add benchmark suite#92
Merged
Merged
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a dedicated benchmarking setup for ContinuousDPs.jl so performance-focused changes can be measured and compared consistently (standalone runs or via PkgBenchmark).
Changes:
- Added a
BenchmarkTools.jl-styleSUITEcovering kernel-level operations and end-to-endsolveruns for representative 1D/2D models. - Introduced a dedicated benchmark environment (
benchmark/Project.toml) to keep benchmark dependencies isolated. - Documented how to run and compare benchmarks (standalone and via PkgBenchmark) in
benchmark/README.md.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| benchmark/benchmarks.jl | Defines the benchmark SUITE with model cases and per-kernel / end-to-end benchmarks, plus standalone execution support. |
| benchmark/Project.toml | Adds a dedicated benchmark environment with the required dependencies. |
| benchmark/README.md | Provides usage instructions for running the suite and comparing commits with PkgBenchmark. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a benchmark suite for the package, so that performance-related PRs can
report standardized before/after comparisons.
benchmark/benchmarks.jl: definesSUITEin the standard BenchmarkTools.jlformat, compatible with PkgBenchmark.jl (and AirspeedVelocity.jl, if we later
add CI benchmarking)
benchmark/Project.toml: dedicated benchmark environmentbenchmark/README.md: usage instructions (standalone and via PkgBenchmark),modeled on GameTheory.jl's
benchmark/README.mdWhat is benchmarked
Three model cases — 1-D stochastic optimal growth with Chebyshev and cubic
spline bases, and the 2-D Santos (1999) model from
test/test_cdp_multidim.jlwith a quadratic spline basis — each with seven benchmarks:
s_wise_max_one_state_s_wise_max!bellman_operatorbellman_operator!compute_greedycompute_greedy!evaluate_policyevaluate_policy!set_eval_nodessolve_PFIsolvewith PFIsolve_VFI_50itersolvewith VFI, capped at 50 iterationsKernel benchmarks use basis coefficients from the converged PFI solution so
inputs are realistic; VFI is capped at 50 iterations so the benchmark measures
a fixed amount of work independently of convergence behavior.
Baseline (Julia 1.12.6, Apple M4 Pro)
Selected results from a verified
PkgBenchmark.benchmarkpkgrun, to indicatecurrent allocation behavior:
["1d_cheb", "s_wise_max_one_state"]["1d_cheb", "bellman_operator"]["1d_cheb", "solve_VFI_50iter"]["2d_spline", "evaluate_policy"]["2d_spline", "solve_VFI_50iter"]Usage
or standalone:
julia --project=benchmark benchmark/benchmarks.jl(see
benchmark/README.mdfor details).Verification
benchmarkpkg("ContinuousDPs")(~1.5 min)--project=benchmark🤖 Generated with Claude Code (Claude Fable 5)