Parallel execution and caching

How do you scale research beyond one core and one machine's memory? VBT splits work into chunks, runs them across threads, processes, or a Ray cluster, and reuses completed results. You control how much work runs at once and what each run keeps, so parallel backtesting in Python can scale to millions of combinations.

Start with your own backtest function

Keep data preparation, signals, and portfolio analysis in one function, then let VBT run it for each parameter combination. This small example tests three entry thresholds in threads and returns one metric per threshold:

close = pd.Series([100.0, 102.0, 101.0, 104.0, 103.0, 106.0])

@vbt.parameterized(merge_func="concat", engine="threadpool", chunk_len=2, show_progress=False)
def test_entry_price(close, entry_price):
    pf = vbt.PF.from_signals(
        close,
        entries=close.vbt.crossed_above(entry_price),
        exits=close.vbt.crossed_below(entry_price),
        fees=0.001,
    )
    return pf.total_return

result = test_entry_price(close, vbt.Param([100.0, 102.0, 104.0], name="entry_price"))
result.round(4)

entry_price
100.0    0.0000
102.0    0.0182
104.0   -0.0010
dtype: float64

Here, chunk_len=2 submits two parameter calls at a time. Each call returns total return as a fraction, and VBT labels the combined result with its parameter value. Set engine="serial" to run the same function sequentially. Scale this pattern to larger tasks and search spaces, as the ten-million-combination example below demonstrates. See strategy optimization for larger search spaces.

Ten million combinations with compact results

close = vbt.YFData.pull("BTC-USD", start="2020-01-01", end="2025-01-01").close

@vbt.parameterized(
    merge_func="row_stack",
    engine="threadpool",  
    chunk_len="auto",
    warmup=True,
)
@njit(nogil=True)
def best_stop_nb(close, fast_window, slow_window, sl_stops):
    fast = vbt.nb.rolling_mean_1d_nb(close, fast_window)
    slow = vbt.nb.rolling_mean_1d_nb(close, slow_window)
    entries = vbt.nb.crossed_above_1d_nb(fast, slow)
    exits = vbt.nb.crossed_above_1d_nb(slow, fast)
    n = len(sl_stops)
    sim_out = vbt.pf_nb.from_signals_nb(
        target_shape=(len(close), n),
        group_lens=np.full(n, 1),
        close=close,
        long_entries=entries,
        long_exits=exits,
        sl_stop=sl_stops.reshape(1, -1),
        fees=0.001,
        save_returns=True,
    )
    sharpe = vbt.ret_nb.sharpe_ratio_nb(sim_out.in_outputs.returns, 365.0)
    best = 0
    for i in range(n):  
        if sharpe[i] > sharpe[best]:
            best = i
    return np.array([sharpe[best], sl_stops[best]])

sl_stops = np.linspace(0.001, 0.30, 1000)
best = best_stop_nb(
    close.values,
    vbt.Param(np.arange(2, 102), name="fast_window"),
    vbt.Param(np.arange(105, 305, 2), name="slow_window"),
    sl_stops,
)
best = best.unstack().set_axis(["sharpe", "sl_stop"], axis=1)
best.shape

(10000, 2)

best.nlargest(3, "sharpe").round(3)

                         sharpe  sl_stop
fast_window slow_window
2           107           1.612    0.011
            119           1.603    0.023
            117           1.594    0.050

The same sweep ran with engine="threadpool" and engine="pathos", which sends chunks to separate processes. Measured on an Apple M3 with 8 cores, both found the same results:

The threaded run peaked at 802 MB of memory, and importing VBT and loading the data alone took 412 MB, so 10,000,000 backtests added about 390 MB: each worker holds one chunk of 1,000 backtests at a time, and only two numbers per window pair survive. Processes won here even though each worker started fresh and compiled the function itself. The faster engine depends on the workload, so time a small slice with each before a long run.

Splitting work into chunks

Chunking splits the arguments of a function into pieces, calls the function on each piece, and merges the results. @vbt.parameterized chunks parameter combinations, @vbt.chunked chunks any array along any axis, and most compiled functions in VBT, including the simulators, come with chunking rules for every argument, so chunked="threadpool" on a portfolio method splits its columns without extra code, as the Chunking highlight below shows. Chunks can nest: an outer loop over batches controls memory while an inner loop runs each batch in parallel.

Choose the split around independent research tasks, such as parameter combinations or portfolios. The built-in simulation rules keep shared-cash groups together, and record merging adjusts column indexes so orders still belong to the right asset or strategy. This lets you keep full records when you need trade-level analysis, or reduce each chunk to metrics when you only need rankings.

Execution engines

vbt.execute runs a list of tasks on one of several engines, chosen by name:

The execution framework provides shared controls for chunk sizes, progress bars, clearing caches or collecting garbage after each chunk, and hooks that run before and after each chunk. Keep worker functions and their dependencies available to the workers, for example by defining them in an importable module. Starting workers can involve importing VBT and compiling callbacks, so give each process many combinations, for example with distribute="chunks", rather than one at a time. Pick one level of parallelism: Numba or Rust parallel=True inside tasks that already run in a thread pool or in Dask makes the threads compete for the same cores. See the Multithreading and Multiprocessing highlights below.

From a laptop to a Ray cluster

You can use the same parameterized function with a local engine or connect Ray to an existing cluster through init_kwargs. VBT submits the work through Ray, while Ray manages the cluster. Batch enough calculations into each task to make worker startup and data transfer worthwhile. This is useful for independent strategy trials, data-processing jobs, or validation runs that outgrow one machine.

For repeated jobs, an existing vbt.Executor can be passed back to vbt.execute to reuse its configuration. Execution hooks also let you prepare shared resources or save summaries around chunks, so the same framework can handle a complete research pipeline.

Reusing results

Portfolios, indicators, and records cache the results of their properties and methods, so pf.stats() computes trades or drawdowns once and reuses them. A central registry tracks every cache, reports hits, misses, and sizes, and clears them by object or class, as the Caching highlight below shows. For long jobs, cache_chunks=True saves each finished chunk to disk, so a restarted run can reuse matching completed chunks, as the Chunk caching highlight below shows.

These serve different needs. Object caches speed up repeated analysis in the current process. Chunk caches preserve completed work across restarts. Numba's compilation cache reuses compiled code and is covered under compute backends. You can inspect and clear selected object caches, or disable caching around a one-off calculation when keeping its results would only use more memory.

Memory

Start with a small batch, then choose chunk size and worker count together. Several workers each holding a chunk can use more memory than one serial worker. Returning only the metrics you need also keeps the final merged result small, as the ten-million-combination example demonstrates.

• Reduce inside the function: return a metric or a summary instead of a portfolio, as the example above does.
• Limit records: max_order_records and max_log_records size the record arrays a simulation allocates when you know how many orders a column can produce.
• Skip results: returning vbt.NoResult from a function leaves that combination out of the merged result, and skip_errored=True does the same for combinations that raise an error. raise_no_results=False returns nothing instead of raising when every combination was skipped.
• Drop what you do not need: pf.replace(open=None, high=None, low=None) keeps a stored portfolio from carrying three price arrays, at the cost of charts and trade statistics that read them, such as the maximum favorable and adverse excursion.
• Free memory between chunks: clear_cache and collect_garbage in the execution options run after each chunk. For process-based jobs, clean up inside the worker that owns the memory. Engines such as Pathos also support recycling workers after a chosen number of tasks.

✅ Most workflows that use the VBT's execution framework, including data pulling, chunking, parameterization, splitting, and optimization, can now offload intermediate results to disk and reload them if the workflow crashes and restarts. You can confidently test billions of parameter combinations on cloud instances without worrying about losing your data.

np.random.seed(42)

@vbt.parameterized(cache_chunks=True, chunk_len=1)  
def basic_iterator(i):
    print("i:", i)
    rand_number = np.random.uniform()
    if rand_number < 0.2:
        print("failed ⛔")
        raise ValueError
    return i

attempt = 0
while True:
    attempt += 1
    print("attempt", attempt)
    try:
        basic_iterator(vbt.Param(np.arange(10)))
        print("completed 🎉")
        break
    except ValueError:
        pass

attempt 1
i: 0
i: 1
i: 2
i: 3
i: 4
failed ⛔
attempt 2
i: 4
failed ⛔
attempt 3
i: 4
failed ⛔
attempt 4
i: 4
i: 5
i: 6
i: 7
failed ⛔
attempt 5
i: 7
i: 8
i: 9
completed 🎉

✅ A new, innovative chunking mechanism lets you specify how arguments should be chunked. It automatically splits array-like arguments, passes each chunk to the function for execution, and then merges the results. This enables you to split large arrays and run any function in a distributed manner. VBT also features a central registry and provides chunking specifications for all arguments of most Numba-compiled functions, including simulation functions. Chunking can be enabled with a single command. Process a large search in smaller pieces that fit your available memory! 🎉

@vbt.chunked(
    chunk_len=100,
    merge_func="concat",  
    execute_kwargs=dict(  
        clear_cache=True,
        collect_garbage=True
    )
)
def backtest(data, fast_windows, slow_windows):  
    fast_ma = vbt.MA.run(data.close, fast_windows, short_name="fast")
    slow_ma = vbt.MA.run(data.close, slow_windows, short_name="slow")
    entries = fast_ma.ma_crossed_above(slow_ma)
    exits = fast_ma.ma_crossed_below(slow_ma)
    pf = vbt.PF.from_signals(data.close, entries, exits)
    return pf.total_return

param_product = vbt.combine_params(  
    dict(
        fast_window=vbt.Param(range(2, 100), condition="fast_window < slow_window"),
        slow_window=vbt.Param(range(2, 100)),
    ),
    build_index=False
)
backtest(
    vbt.YFData.pull(["BTC-USD", "ETH-USD"]),  
    vbt.Chunked(param_product["fast_window"]),  
    vbt.Chunked(param_product["slow_window"])
)

fast_window  slow_window  symbol
2            3            BTC-USD    193.124482
                          ETH-USD     12.247315
             4            BTC-USD    159.600953
                          ETH-USD     15.825041
             5            BTC-USD    124.703676
                                            ...
97           98           ETH-USD      3.947346
             99           BTC-USD     25.551881
                          ETH-USD      3.442949
98           99           BTC-USD     27.943574
                          ETH-USD      3.540720
Name: total_return, Length: 9506, dtype: float64

✅ Integration of ThreadPoolExecutor from concurrent.futures, ThreadPool from pathos, and Dask backend for running multiple chunks across multiple threads. This is best for speeding up heavyweight functions that release the GIL, such as Numba and C functions. Multithreading + Chunking + Numba = 💪

data = vbt.YFData.pull(["BTC-USD", "ETH-USD"])

%timeit vbt.PF.from_random_signals(data.close, n=[100] * 1000, seed=42)

613 ms ± 37.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit vbt.PF.from_random_signals(data.close, n=[100] * 1000, seed=42, chunked="threadpool")

294 ms ± 8.91 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

✅ Integration of ProcessPoolExecutor from concurrent.futures, ProcessPool and ParallelPool from pathos, WorkerPool from mpire, and Ray backend for running multiple chunks across multiple processes. This is best for speeding up heavyweight functions that do not release the GIL, such as regular Python functions, as well as lightweight arguments that are easy to serialize. Ever wanted to test billions of hyperparameter combinations across multiple machines? Scale functions and entire applications in the cloud using Ray clusters 👀

@vbt.chunked(
    size=vbt.ArraySizer(arg_query="items", axis=1),
    arg_take_spec=dict(
        items=vbt.ArraySelector(axis=1)
    ),
    merge_func=np.column_stack
)
def bubble_sort(items):
    items = items.copy()
    for i in range(len(items)):
        for j in range(len(items) - 1 - i):
            if items[j] > items[j + 1]:
                items[j], items[j + 1] = items[j + 1], items[j]
    return items

np.random.seed(42)
items = np.random.uniform(size=(1000, 3))

%timeit bubble_sort(items)

456 ms ± 1.36 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit bubble_sort(items, _execute_kwargs=dict(engine="pathos"))

165 ms ± 1.51 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

✅ Caching has been completely reimplemented and is now managed by a central registry. This enables tracking useful statistics for all cacheable parts of VBT, such as showing the total cached size in MB. You get full control and transparency 🪟

data = vbt.YFData.pull("BTC-USD")
pf = vbt.PF.from_random_signals(data.close, n=5, seed=42)
_ = pf.stats()

pf.get_ca_setup().get_status_overview(
    filter_func=lambda setup: setup.caching_enabled,
    include=["hits", "misses", "total_size"]
)

                                 hits  misses total_size
object
portfolio:0.drawdowns               0       1    70.9 kB
portfolio:0.exit_trades             0       1    70.5 kB
portfolio:0.filled_close            6       1    24.3 kB
portfolio:0.init_cash               3       1   32 Bytes
portfolio:0.init_position           0       1   32 Bytes
portfolio:0.init_position_value     0       1   32 Bytes
portfolio:0.init_value              5       1   32 Bytes
portfolio:0.input_value             1       1   32 Bytes
portfolio:0.orders                  9       1    69.7 kB
portfolio:0.total_profit            1       1   32 Bytes
portfolio:0.trades                  0       1    70.5 kB