Benchmarks

This directory contains generated benchmark reports. The benchmark tooling lives in vectorbtpro.benchmarks so it can be shipped with the package.

The benchmarks are correctness-aware microbenchmarks for registered jitting backends. They use deterministic inputs, can verify output parity, and emit CSV for targeted runs or Markdown/JSON reports for the full matrix.

Requirements

Install the optional dependencies required by every backend you select. For compiled backends, use release builds; debug builds include extra overhead and are not representative.

For the Rust backend:

python -m maturin develop --manifest-path rust/Cargo.toml --release

Runs that select the Rust backend require vectorbtpro-rust to be installed and version-compatible.

Help

Use the CLI help as the source of truth for available options:

python -m vectorbtpro.benchmarks.bench_engine_cli --help
python -m vectorbtpro.benchmarks.bench_matrix_cli --help

Targeted runs

Use the benchmark engine when you want a focused CSV result for one input model and a small set of cases:

python -m vectorbtpro.benchmarks.bench_engine_cli \
  --backend nb \
  --backend rs \
  --input-model 1d \
  --pattern returns

--input-model selects 0d, 1d, or 2d benchmark inputs. --pattern filters benchmark case names by substring. Prefix a pattern with = to match one complete case name.

The same engine is available programmatically:

from vectorbtpro import *

results = vbt.run_benchmarks(
    input_model="1d",
    backend_ids=("nb", "rs"),
    patterns=["returns"]
)
print(results)

Matrix reports

Use the matrix runner for Markdown reports and the raw JSON data used to render them:

python -m vectorbtpro.benchmarks.bench_matrix_cli --backend nb --backend rs

Use --full for the standard matrix:

python -m vectorbtpro.benchmarks.bench_matrix_cli --full

This expands to nb, rs, rs_raw, ab, ab_raw, abm, and abm_raw, replaces the matrix, and writes speedup reports for wrapped and raw variants against nb.

ab is AutoBench and selects among concrete backends inside the requested Serial or Parallel mode. ab_raw is AutoBench using raw candidate timings and raw prepared backend calls. abm is AutoBenchMixed and selects among serial and parallel concrete kernels, so it is reported once in a Mixed section. abm_raw is the raw AutoBenchMixed variant. The matrix runner benchmarks concrete backends first, then uses those timings for AutoBench in the same run. Auto-only replacement runs are rejected.

Reports are organized as Serial and Parallel sections, each split into 0D kernels, 1D kernels, and 2D kernels. Runtime reports include per-configuration statistics, and multi-shape sections also include statistics grouped by total element count. Generated configurations are capped at one million input elements. AutoBenchMixed reports use a single Mixed section instead of separate Serial and Parallel sections.

By default, run_bench_matrix runs the benchmark engine in subprocesses for isolation. Pass use_subprocess=False to run benchmarks in the same Python process.

Use --pattern for a targeted refresh. The default write mode comes from the benchmark settings; use --update to preserve unrelated rows from the existing JSON cache, or --replace to write only the current run. Use --from-cache to regenerate reports from cached JSON without rerunning benchmarks.

Use --run-backend to rerun only selected backends while keeping all --backend reports in the matrix. This is useful when only one backend changed but speedup reports should be refreshed from mixed cached and new timings:

python -m vectorbtpro.benchmarks.bench_matrix_cli \
    --backend nb --backend rs \
    --run-backend rs \
    --update \
    --pattern <subpackage-or-function>

Interpretation

For generated X_VS_Y speedup reports, X is the faster-is-better side of the comparison and Y is the baseline side. Each value is runtime(Y) / runtime(X), so values above 1.00x mean X was faster for that case, and values below 1.00x mean Y was faster. Explicit --speedup-pair FIRST:SECOND options are ordered, so --backend order does not control speedup direction. - means the value is unavailable for that backend or mode.

For abm comparisons, the concrete backend side uses the faster available Serial or Parallel runtime for the same case and configuration. For example, --speedup-pair abm:nb writes ABM_VS_NB.md, where values above 1.00x mean AutoBenchMixed was faster than the best Numba mode.

Absolute runtimes matter, especially for tiny kernels where a large speedup may still represent only nanoseconds or microseconds. Benchmark numbers are sensitive to CPU, OS scheduling, Python and NumPy versions, backend versions, compiler versions, and whether compiled extensions were built in release mode. Use the generated environment block when publishing or comparing results across machines.

New benchmark cases

Add a small case-builder function in the nearest bench_cases.py module to the code being benchmarked. The benchmark engine discovers these modules automatically and sorts generated cases by benchmark name.

Keep benchmark cases deterministic, representative of public dispatch behavior, cheap enough to run across the matrix, and explicit about cases where parity cannot be exact.

After adding or changing cases, run at least one targeted benchmark:

python -m vectorbtpro.benchmarks.bench_engine_cli \
  --backend nb \
  --backend rs \
  --input-model <0d|1d|2d> \
  --pattern <subpackage-or-function>

Then regenerate the matrix reports: