Financial data pipelines in Python

Feature-oriented data

New in1.14.0

✅ The main limitation of VBT's data class was that it could only store data in a symbol-oriented format. This meant features such as OHLC had to be combined into a single DataFrame beforehand. This approach can be somewhat counterproductive, as in VBT, we typically work with these features separately. For example, when calling data.close, VBT scans for "close" columns across all symbols, extracts them, and concatenates them into another DataFrame. To address this, the data class has been redesigned to natively support storing feature-oriented data as well.

Create a feature-oriented data instance from various portfolio time series

data = vbt.YFData.pull(["AAPL", "MSFT", "GOOG"], tz_convert="UTC")
pf = data.run("from_random_signals", n=[10, 20, 30], seed=42)

pf_data = vbt.Data.from_data(
    vbt.feature_dict({
        "cash": pf.cash,
        "assets": pf.assets,
        "asset_value": pf.asset_value,
        "value": pf.value
    })
)
pf_data.get(feature="cash", symbol=(10, "AAPL"))

Date
1980-12-12 05:00:00+00:00      100.000000
1980-12-15 05:00:00+00:00      100.000000
1980-12-16 05:00:00+00:00      100.000000
1980-12-17 05:00:00+00:00      100.000000
1980-12-18 05:00:00+00:00      100.000000
                                      ...
2023-08-25 04:00:00+00:00     1837.648268
2023-08-28 04:00:00+00:00     1837.648268
2023-08-29 04:00:00+00:00     1837.648268
2023-08-30 04:00:00+00:00     1837.648268
2023-08-31 04:00:00+00:00     1837.648268
Name: (10, AAPL), Length: 10770, dtype: float64

Parallel data

New in1.10.0

✅ Data fetching and updating can be easily parallelized.

Benchmark fetching multiple symbols serially and concurrently

symbols = ["SPY", "TLT", "XLF", "XLE", "XLU", "XLK", "XLB", "XLP", "XLY", "XLI", "XLV"]

with vbt.Timer() as timer:
    data = vbt.YFData.pull(symbols)
print(timer.elapsed())

4.52 seconds

with vbt.Timer() as timer:
    data = vbt.YFData.pull(symbols, execute_kwargs=dict(engine="threadpool"))
print(timer.elapsed())

918.54 milliseconds

Runnable data

New in1.5.0

✅ Tired of figuring out which arguments are required by an indicator? Data instances can now recognize the arguments of any indicator or function, map them to column names, and run the function by passing in the required columns. You can also change the mapping, override indicator parameters, and query indicators by name. The data instance will search all integrated indicator packages and return the first (and best) match it finds.

Run Stochastic RSI by data

data = vbt.YFData.pull("BTC-USD")
stochrsi = data.run("stochrsi")
stochrsi.fastd

Date
2014-09-17 00:00:00+00:00          NaN
2014-09-18 00:00:00+00:00          NaN
2014-09-19 00:00:00+00:00          NaN
2014-09-20 00:00:00+00:00          NaN
2014-09-21 00:00:00+00:00          NaN
                                   ...
2023-01-15 00:00:00+00:00    96.168788
2023-01-16 00:00:00+00:00    91.733393
2023-01-17 00:00:00+00:00    78.295255
2023-01-18 00:00:00+00:00    48.793133
2023-01-20 00:00:00+00:00    26.242474
Name: Close, Length: 3047, dtype: float64

Data parsing

New in1.3.0

✅ Tired of passing open, high, low, and close as separate time series? Portfolio class methods now accept a data instance instead of just close and automatically extract the contained OHLC data. This small but handy feature saves you time!

Backtest random signals from a data instance

data = vbt.YFData.pull("BTC-USD", start="2020-01", end="2020-03")
pf = vbt.PF.from_random_signals(data, n=10, seed=42)

Data transformation

New in1.2.3

✅ After fetching data, how do you change it? There is a new method that puts all symbols into a single DataFrame and passes it to a UDF for transformation.

Remove weekends

data = vbt.YFData.pull(["BTC-USD", "ETH-USD"], start="2020-01-01", end="2020-01-14")
new_data = data.transform(lambda df: df[~df.index.weekday.isin([5, 6])])
new_data.close

symbol                         BTC-USD     ETH-USD
Date
2020-01-01 00:00:00+00:00  7200.174316  130.802002
2020-01-02 00:00:00+00:00  6985.470215  127.410179
2020-01-03 00:00:00+00:00  7344.884277  134.171707
2020-01-06 00:00:00+00:00  7769.219238  144.304153
2020-01-07 00:00:00+00:00  8163.692383  143.543991
2020-01-08 00:00:00+00:00  8079.862793  141.258133
2020-01-09 00:00:00+00:00  7879.071289  138.979202
2020-01-10 00:00:00+00:00  8166.554199  143.963776
2020-01-13 00:00:00+00:00  8144.194336  144.226593

Synthetic OHLC

New in1.2.1

✅ New basic models are available for generating synthetic OHLC data. These are especially useful for leakage detection.

Generate 3 months of synthetic data using Geometric Brownian Motion

data = vbt.GBMOHLCData.pull("R", start="2022-01", end="2022-04", seed=42)
data.plot().show()

Data merging

New in1.1.0

✅ Often, there is a need to backtest symbols from different exchanges by placing them in the same basket. For this purpose, VBT offers a class method that can merge multiple data instances into a single one. You can not only combine multiple symbols, but also merge datasets for a single symbol—all done automatically!

Pull BTC datasets from various exchanges and plot the last full year relative to their mean

binance_data = vbt.CCXTData.pull("BTCUSDT", exchange="binance")
bybit_data = vbt.CCXTData.pull("BTCUSDT", exchange="bybit")
bitfinex_data = vbt.CCXTData.pull("BTC/USDT", exchange="bitfinex")
kucoin_data = vbt.CCXTData.pull("BTC-USDT", exchange="kucoin")

data = vbt.Data.merge([
    binance_data.rename({"BTCUSDT": "Binance"}),
    bybit_data.rename({"BTCUSDT": "Bybit"}),
    bitfinex_data.rename({"BTC/USDT": "Bitfinex"}),
    kucoin_data.rename({"BTC-USDT": "KuCoin"}),
], missing_index="drop", silence_warnings=True)

@njit
def rescale_nb(x):
    return (x - x.mean()) / x.mean()

rescaled_close = data.close.vbt.row_apply(rescale_nb)
rescaled_close = rescaled_close.vbt.rolling_mean(30)
last_full_year = rescaled_close.index[-1].year - 1
rescaled_close.loc[str(last_full_year)].vbt.plot().show()