Data storage and databases

Where should market data live once it is downloaded, and how do you read back just the part you need? VBT saves any data object to CSV, HDF5, Feather, Parquet, DuckDB, SQL databases, or ArcticDB, and reads it back into a data object ready for research. Store market data in Python and select the dates and symbols you need for your next backtest.

data = vbt.GBMOHLCData.pull(  
    ["AAA", "BBB", "CCC"],
    start="2024-01-01",
    end="2024-01-29",
    timeframe="1min",
    std=0.0005,
    seed=42,
)
data.loc[:"2024-01-21 23:59"].to_duckdb(connection="market.duckdb")
data.loc["2024-01-22":].to_duckdb(  
    connection="market.duckdb",
    if_exists="append",
)
vbt.DuckDBData.list_tables(connection="market.duckdb")

['AAA', 'BBB', 'CCC']

week = vbt.DuckDBData.pull(
    ["AAA", "BBB", "CCC"],
    start="2024-01-15",
    end="2024-01-22",
    tz="UTC",  
    connection="market.duckdb",
)
week.close.shape

(10080, 3)

week.close.head(3).round(2)

symbol                       AAA     BBB    CCC
2024-01-15 00:00:00+00:00  67.01  176.99  60.39
2024-01-15 00:01:00+00:00  66.99  176.51  60.20
2024-01-15 00:02:00+00:00  67.18  175.62  60.15

Each symbol becomes a table, appends extend it, and a query returns only the requested week as a regular data object, ready for indicators and backtests.

Choosing a storage format

Choose around how you collect and reuse the data:

You can use more than one. Keep a Parquet archive, query a subset with DuckDB, and pass the result into a backtest. The examples below show both storage and retrieval.

Files

Every data object has to_csv, to_hdf, to_feather, and to_parquet, and the matching classes vbt.CSVData, vbt.HDFData, vbt.FeatherData, and vbt.ParquetData read the files back. The same four weeks of minute bars take very different space depending on the format:

import os

data.to_csv("csv")
data.to_hdf("data.h5")
data.to_parquet("parquet")
data.to_feather("feather")

def size_mb(path):
    if os.path.isfile(path):
        return os.path.getsize(path) / 1e6
    return sum(
        os.path.getsize(os.path.join(d, f))
        for d, _, files in os.walk(path)
        for f in files
    ) / 1e6

paths = ["csv", "data.h5", "parquet", "feather", "market.duckdb"]
pd.Series({path: round(size_mb(path), 1) for path in paths})

csv              12.0
data.h5           5.2
parquet           5.9
feather           4.8
market.duckdb     5.8
dtype: float64

CSV is readable everywhere but is the largest and slowest to parse. HDF5 keeps many symbols in one file. Parquet is compressed, columnar, and read by most data tools, and it can be partitioned by symbol or date. Reading a folder finds the files in it, matching symbols by file name. CSV and HDF readers can process chunks and keep only the rows you need. Parquet filters select dates or other values, as the partitioned dataset example below shows.

With separate files, each file is treated as one symbol. A partitioned Parquet directory can instead hold many date partitions for the same symbol. For an existing collection of daily files, combine them with Pandas and use vbt.Data.from_data, or query them with DuckDB.

Databases

vbt.DuckDBData stores and queries data in DuckDB, an embedded analytical database that runs in your process and can also query Parquet and CSV files directly. vbt.SQLData works with any database that SQLAlchemy supports, such as PostgreSQL, MySQL, or SQLite, and vbt.ArcticDBData uses ArcticDB for large versioned time-series stores. DuckDB and SQLData accept custom SQL queries and can list schemas, tables, and views. ArcticDBData lists libraries and symbols, and reads through ArcticDB's own API. vbt.DuckDBData.pull opens the database read-only by default, so several research processes can read the same file at once. Connections can be passed in as objects or created from a URL. For repeated SQL reads and updates, you can reuse your SQLAlchemy engine and its connection pool.

ArcticDBData can select a stored version, date range, and columns. This lets you revisit a saved dataset version when comparing research runs, even after newer data has been written.

Use your existing data store

You do not need a separate table for every symbol. A custom query can select rows from a shared table, join other inputs, and rename columns for VBT. The result becomes the same kind of data object as a provider download.

This small example creates a Parquet file with made-up prices for two symbols, then reads only the last two days. In your own archive, the file already exists:

archive_rows = pd.DataFrame(
    {
        "timestamp": pd.to_datetime(["2024-01-01", "2024-01-02", "2024-01-03"] * 2, utc=True),
        "ticker": ["AAA"] * 3 + ["BBB"] * 3,
        "price": [100.0, 102.0, 104.0, 50.0, 49.0, 51.0],
    }
)
archive_rows.to_parquet("archive.parquet", index=False)
archive_data = vbt.DuckDBData.pull(
    ["AAA", "BBB"],
    query=vbt.key_dict(
        {
            symbol: f"""
                SELECT timestamp, price AS Close
                FROM read_parquet('archive.parquet')
                WHERE ticker = '{symbol}' AND timestamp >= '2024-01-02'
                ORDER BY timestamp
            """
            for symbol in ["AAA", "BBB"]
        }
    ),
    index_col="timestamp",
)
archive_data.close

symbol                       AAA   BBB
timestamp
2024-01-02 00:00:00+00:00  102.0  49.0
2024-01-03 00:00:00+00:00  104.0  51.0

The SQL query selects the rows and columns before they become a Pandas object. DuckDB can filter Parquet data while reading, so a research run can work with the relevant slice of a larger archive. SQLData offers the same custom-query entry point for your SQL database. If you already have a loader that returns DataFrames, the market data page shows how to bring those into VBT too.

Keeping stores current

Keep your research data current automatically. Data savers combine update with saving: they pull whatever is new and append it to the store on a schedule, and they can be stopped and resumed without losing their place, as the Data saver highlight below shows. During research, data caching stores each download locally so that rerunning a notebook reads the cache instead of the provider, as in the Data caching highlight.

Savers are available for CSV, HDF5, SQL, DuckDB, and ArcticDB, and work with multiple symbols. Use the saver matching your store, such as vbt.DuckDBDataSaver or vbt.SQLDataSaver, with a provider data object that knows how to fetch updates. The data pipelines page covers incremental updates and scheduling.

Save a research session

To keep the data together with its VBT settings and fetch arguments, save the whole object with data.save and restore it with the data class's load method. This is useful when you want to continue working with the same provider object in another session. Tabular exports such as Parquet are useful for sharing the underlying data with other tools. Configuration and persistence covers saving and restoring VBT objects.

✅ Data caching is now available for all data classes, allowing you to cache data on disk when pulling from a (remote) data source. Caching is performed using LMDB, which provides efficient key-value storage. The cache key is generated based on the hash of the class, symbol, and other fetch parameters, ensuring that different data pulls are cached separately.

data = vbt.TVData.pull(
    "AAPL",
    exchange="NASDAQ",
    timeframe="1 minute",
    tz="America/New_York",
    cache=True  
)

✅ ArcticDB is a serverless DataFrame database engine designed for the Python Data Science ecosystem. It can store, retrieve and process Series and DataFrames at scale, backed by commodity object storage (S3-compatible storages and Azure Blob Storage). ArcticDB requires zero additional infrastructure beyond a running Python environment and access to object storage and can be installed in seconds.

data = vbt.TVData.pull(
    "AAPL",
    exchange="NASDAQ",
    timeframe="1 minute",
    tz="America/New_York"
)

data.to_arcticdb()

day_data = vbt.ArcticDBData.pull(
    "AAPL",
    start="2026-02-27 09:30:00",
    end="2026-02-27 16:00:00",
    tz="America/New_York",
)
day_data.get()

                             Open    High     Low    Close    Volume
datetime
2026-02-27 09:30:00-05:00  272.90  272.90  269.80  269.960   49256.0
2026-02-27 09:31:00-05:00  269.97  270.12  269.19  269.230   43715.0
2026-02-27 09:32:00-05:00  269.18  270.05  269.18  269.890   33913.0
2026-02-27 09:33:00-05:00  269.85  270.01  269.60  269.770   27569.0
2026-02-27 09:34:00-05:00  269.74  269.98  269.40  269.500   22963.0
...                           ...     ...     ...      ...       ...
2026-02-27 15:55:00-05:00  264.41  264.51  263.15  263.490  162153.0
2026-02-27 15:56:00-05:00  263.51  263.78  263.23  263.670  114741.0
2026-02-27 15:57:00-05:00  263.67  263.78  262.89  263.655  113641.0
2026-02-27 15:58:00-05:00  263.61  263.92  263.25  263.620  158462.0
2026-02-27 15:59:00-05:00  263.68  264.28  263.65  264.200  107619.0

[390 rows x 5 columns]

✅ Thanks to DuckDB, you can now run SQL queries directly on data instances.

data = vbt.TVData.pull(
    "AAPL",
    exchange="NASDAQ",
    timeframe="1 minute",
    tz="America/New_York"
)

data.sql("""
    SELECT datetime, AVG(Close) OVER(
        ORDER BY "datetime" ASC
        RANGE BETWEEN INTERVAL 14 DAYS PRECEDING AND CURRENT ROW
    ) AS "Moving Average"
    FROM "AAPL";
""")

datetime
2023-09-11 09:30:00-04:00    180.080000
2023-09-11 09:31:00-04:00    179.965000
2023-09-11 09:32:00-04:00    180.000000
2023-09-11 09:33:00-04:00    180.022500
2023-09-11 09:34:00-04:00    179.984000
                                    ...
2023-10-20 15:55:00-04:00    177.786669
2023-10-20 15:56:00-04:00    177.785492
2023-10-20 15:57:00-04:00    177.784322
2023-10-20 15:58:00-04:00    177.783166
2023-10-20 15:59:00-04:00    177.781986
Name: Moving Average, Length: 11700, dtype: float64

✅ DuckDB is a high-performance analytical database system that offers a robust SQL dialect for interacting with various data stores. Not only can it run analytical queries on local data, even if the data does not fit into memory and without needing a distributed framework, but it can also query CSV, Parquet, and JSON files directly.

data = vbt.TVData.pull(
    "AAPL",
    exchange="NASDAQ",
    timeframe="1 minute",
    tz="America/New_York"
)

URL = "database.duckdb"
data.to_duckdb(connection=URL)

day_data = vbt.DuckDBData.pull(
    "AAPL",
    start="2023-10-02 09:30:00",
    end="2023-10-02 16:00:00",
    tz="America/New_York",
    connection=URL
)
day_data.get()

                              Open    High     Low   Close    Volume
datetime
2023-10-02 09:30:00-04:00  171.260  171.34  170.93  171.10   61654.0
2023-10-02 09:31:00-04:00  171.130  172.37  171.13  172.30   53481.0
2023-10-02 09:32:00-04:00  172.310  172.64  172.16  172.64   44750.0
2023-10-02 09:33:00-04:00  172.640  172.97  172.54  172.78   53195.0
2023-10-02 09:34:00-04:00  172.780  173.07  172.75  173.00   47416.0
...                            ...     ...     ...     ...       ...
2023-10-02 15:55:00-04:00  173.300  173.51  173.26  173.51   61619.0
2023-10-02 15:56:00-04:00  173.525  173.59  173.42  173.43   45066.0
2023-10-02 15:57:00-04:00  173.430  173.55  173.42  173.50   45220.0
2023-10-02 15:58:00-04:00  173.510  173.60  173.46  173.56   47371.0
2023-10-02 15:59:00-04:00  173.560  173.78  173.56  173.75  161253.0

[390 rows x 5 columns]

✅ SQLAlchemy provides a standard interface that allows developers to create database-agnostic code for communicating with a wide range of SQL database engines. With its help, you can now easily store data in SQL databases and read from them as well.

data = vbt.TVData.pull(
    "AAPL",
    exchange="NASDAQ",
    timeframe="1 minute",
    tz="America/New_York"
)

URL = "postgresql://postgres:postgres@localhost:5432"
data.to_sql(engine=URL)

day_data = vbt.SQLData.pull(
    "AAPL",
    start="2023-10-02 09:30:00",
    end="2023-10-02 16:00:00",
    tz="America/New_York",
    engine=URL
)
day_data.get()

                              Open    High     Low   Close    Volume
datetime
2023-10-02 09:30:00-04:00  171.260  171.34  170.93  171.10   61654.0
2023-10-02 09:31:00-04:00  171.130  172.37  171.13  172.30   53481.0
2023-10-02 09:32:00-04:00  172.310  172.64  172.16  172.64   44750.0
2023-10-02 09:33:00-04:00  172.640  172.97  172.54  172.78   53195.0
2023-10-02 09:34:00-04:00  172.780  173.07  172.75  173.00   47416.0
...                            ...     ...     ...     ...       ...
2023-10-02 15:55:00-04:00  173.300  173.51  173.26  173.51   61619.0
2023-10-02 15:56:00-04:00  173.525  173.59  173.42  173.43   45066.0
2023-10-02 15:57:00-04:00  173.430  173.55  173.42  173.50   45220.0
2023-10-02 15:58:00-04:00  173.510  173.60  173.46  173.56   47371.0
2023-10-02 15:59:00-04:00  173.560  173.78  173.56  173.75  161253.0

[390 rows x 5 columns]

✅ Data can be written to Feather with PyArrow and to Parquet using PyArrow or FastParquet. Parquet performs especially well in write-once, read-many scenarios, providing highly efficient data compression and decompression, making it a great choice for storing time series data.

data = vbt.TVData.pull(
    "AAPL",
    exchange="NASDAQ",
    timeframe="1 minute",
    tz="America/New_York"
)

data.to_parquet(partition_by="day")  

day_data = vbt.ParquetData.pull("AAPL", filters=[("group", "==", "2023-10-02")])
day_data.get()

                              Open    High     Low   Close    Volume
datetime
2023-10-02 09:30:00-04:00  171.260  171.34  170.93  171.10   61654.0
2023-10-02 09:31:00-04:00  171.130  172.37  171.13  172.30   53481.0
2023-10-02 09:32:00-04:00  172.310  172.64  172.16  172.64   44750.0
2023-10-02 09:33:00-04:00  172.640  172.97  172.54  172.78   53195.0
2023-10-02 09:34:00-04:00  172.780  173.07  172.75  173.00   47416.0
...                            ...     ...     ...     ...       ...
2023-10-02 15:55:00-04:00  173.300  173.51  173.26  173.51   61619.0
2023-10-02 15:56:00-04:00  173.525  173.59  173.42  173.43   45066.0
2023-10-02 15:57:00-04:00  173.430  173.55  173.42  173.50   45220.0
2023-10-02 15:58:00-04:00  173.510  173.60  173.46  173.56   47371.0
2023-10-02 15:59:00-04:00  173.560  173.78  173.56  173.75  161253.0

[390 rows x 5 columns]

✅ Imagine a script that can periodically pull the latest data from an exchange and save it to disk, all without your intervention. VBT implements two classes that can do exactly this: one that saves to CSV and another that saves to HDF.

from vectorbtpro import *

import logging
logging.basicConfig(level=logging.INFO)

if __name__ == "__main__":
    if vbt.CSVDataSaver.file_exists():
        csv_saver = vbt.CSVDataSaver.load()
        csv_saver.update()
        init_save = False
    else:
        data = vbt.BinanceData.pull(
            "BTCUSDT",
            start="10 minutes ago UTC",
            timeframe="1 minute"
        )
        csv_saver = vbt.CSVDataSaver(data)
        init_save = True
    csv_saver.update_every(1, "minute", init_save=init_save)
    csv_saver.save()  

$ python BTCUSDT_1m_saver.py
2023-02-01 12:26:36.744000+00:00 - 2023-02-01 12:36:00+00:00: : 1it [00:01,  1.22s/it]
INFO:vectorbtpro.data.saver:Saved initial 10 rows from 2023-02-01 12:27:00+00:00 to 2023-02-01 12:36:00+00:00
INFO:vectorbtpro.utils.schedule_:Starting schedule manager with jobs [Every 1 minute do update(save_kwargs=None) (last run: [never], next run: 2023-02-01 13:37:38)]
INFO:vectorbtpro.data.saver:Saved 2 rows from 2023-02-01 12:36:00+00:00 to 2023-02-01 12:37:00+00:00
INFO:vectorbtpro.data.saver:Saved 2 rows from 2023-02-01 12:37:00+00:00 to 2023-02-01 12:38:00+00:00
INFO:vectorbtpro.utils.schedule_:Stopping schedule manager

$ python BTCUSDT_1m_saver.py
INFO:vectorbtpro.utils.schedule_:Starting schedule manager with jobs [Every 1 minute do update(save_kwargs=None) (last run: [never], next run: 2023-02-01 13:42:08)]
INFO:vectorbtpro.data.saver:Saved 5 rows from 2023-02-01 12:38:00+00:00 to 2023-02-01 12:42:00+00:00
INFO:vectorbtpro.utils.schedule_:Stopping schedule manager

✅ Once you have fetched remote data, you will most likely want to save it to disk. There are two new options for this: you can either serialize the entire data class, or save the actual data to CSV or HDF5. Each dataset can be stored in a single flat file, which makes handling the data easier than using a database. After saving, you can easily load the data back either by deserializing or by using data classes that specialize in loading from CSV and HDF5 files. These classes support a range of features, including filtering by row and datetime ranges, updating, chunking, and even a smart dataset search that can recursively walk through sub-directories and return datasets that match a specific glob pattern or regular expression 🧲

btc_data = vbt.BinanceData.pull("BTCUSDT")
eth_data = vbt.BinanceData.pull("ETHUSDT")

btc_data.to_hdf()
eth_data.to_hdf()

data = vbt.BinanceData.from_hdf(start="2020", end="2021")

data.close

symbol                      BTCUSDT  ETHUSDT
Open time
2020-01-01 00:00:00+00:00   7200.85   130.77
2020-01-02 00:00:00+00:00   6965.71   127.19
2020-01-03 00:00:00+00:00   7344.96   134.35
2020-01-04 00:00:00+00:00   7354.11   134.20
2020-01-05 00:00:00+00:00   7358.75   135.37
...                             ...      ...
2020-12-27 00:00:00+00:00  26281.66   685.11
2020-12-28 00:00:00+00:00  27079.41   730.41
2020-12-29 00:00:00+00:00  27385.00   732.00
2020-12-30 00:00:00+00:00  28875.54   752.17
2020-12-31 00:00:00+00:00  28923.63   736.42

[366 rows x 2 columns]