# Data storage and databases (/features/data/data-storage-and-databases)

## Data caching \[#data-caching]

New in v2026.3.1

✅ Data caching is now available for all data classes, allowing you to cache data on disk when
pulling from a (remote) data source. Caching is performed using LMDB, which provides efficient
key-value storage. The cache key is generated based on the hash of the class, symbol, and other
fetch parameters, ensuring that different data pulls are cached separately.

```python title="Pull AAPL minute data from TradingView or cache if available"
>>> data = vbt.TVData.pull(
...     "AAPL",
...     exchange="NASDAQ",
...     timeframe="1 minute",
...     tz="America/New_York",
...     cache=True  # (1)
... )
```

1.  Next time you pull the same data with the same parameters, it will be loaded from the cache.
    Use `refresh_cache=True` to refresh the cache.

## ArcticDB \[#arcticdb]

New in v2026.3.1

✅ [ArcticDB](https://arcticdb.io/) is a serverless DataFrame database engine designed for
the Python Data Science ecosystem. It can store, retrieve and process Series and DataFrames at scale,
backed by commodity object storage (S3-compatible storages and Azure Blob Storage).
ArcticDB requires zero additional infrastructure beyond a running Python environment and
access to object storage and can be installed in seconds.

```python title="Save minute data to a ArcticDB database, and read one day"
>>> data = vbt.TVData.pull(
...     "AAPL",
...     exchange="NASDAQ",
...     timeframe="1 minute",
...     tz="America/New_York"
... )

>>> data.to_arcticdb()

>>> day_data = vbt.ArcticDBData.pull(
...     "AAPL",
...     start="2026-02-27 09:30:00",
...     end="2026-02-27 16:00:00",
...     tz="America/New_York",
... )
>>> day_data.get()
                             Open    High     Low    Close    Volume
datetime
2026-02-27 09:30:00-05:00  272.90  272.90  269.80  269.960   49256.0
2026-02-27 09:31:00-05:00  269.97  270.12  269.19  269.230   43715.0
2026-02-27 09:32:00-05:00  269.18  270.05  269.18  269.890   33913.0
2026-02-27 09:33:00-05:00  269.85  270.01  269.60  269.770   27569.0
2026-02-27 09:34:00-05:00  269.74  269.98  269.40  269.500   22963.0
...                           ...     ...     ...      ...       ...
2026-02-27 15:55:00-05:00  264.41  264.51  263.15  263.490  162153.0
2026-02-27 15:56:00-05:00  263.51  263.78  263.23  263.670  114741.0
2026-02-27 15:57:00-05:00  263.67  263.78  262.89  263.655  113641.0
2026-02-27 15:58:00-05:00  263.61  263.92  263.25  263.620  158462.0
2026-02-27 15:59:00-05:00  263.68  264.28  263.65  264.200  107619.0

[390 rows x 5 columns]
```

## SQL queries \[#sql-queries]

New in v2023.10.22

✅ Thanks to [DuckDB](https://duckdb.org/), you can now run SQL queries directly on data instances.

```python title="Run a rolling average of 14 days on minute data using SQL"
>>> data = vbt.TVData.pull(
...     "AAPL",
...     exchange="NASDAQ",
...     timeframe="1 minute",
...     tz="America/New_York"
... )

>>> data.sql("""
...     SELECT datetime, AVG(Close) OVER(
...         ORDER BY "datetime" ASC
...         RANGE BETWEEN INTERVAL 14 DAYS PRECEDING AND CURRENT ROW
...     ) AS "Moving Average"
...     FROM "AAPL";
... """)
datetime
2023-09-11 09:30:00-04:00    180.080000
2023-09-11 09:31:00-04:00    179.965000
2023-09-11 09:32:00-04:00    180.000000
2023-09-11 09:33:00-04:00    180.022500
2023-09-11 09:34:00-04:00    179.984000
                                    ...
2023-10-20 15:55:00-04:00    177.786669
2023-10-20 15:56:00-04:00    177.785492
2023-10-20 15:57:00-04:00    177.784322
2023-10-20 15:58:00-04:00    177.783166
2023-10-20 15:59:00-04:00    177.781986
Name: Moving Average, Length: 11700, dtype: float64
```

## DuckDB \[#duckdb]

New in v2023.10.22

✅ [DuckDB](https://duckdb.org/) is a high-performance analytical database system that offers
a robust SQL dialect for interacting with various data stores. Not only can it run analytical queries on
local data, even if the data does not fit into memory and without needing a distributed framework,
but it can also query CSV, Parquet, and JSON files directly.

```python title="Save minute data to a DuckDB database, and read one day"
>>> data = vbt.TVData.pull(
...     "AAPL",
...     exchange="NASDAQ",
...     timeframe="1 minute",
...     tz="America/New_York"
... )

>>> URL = "database.duckdb"
>>> data.to_duckdb(connection=URL)

>>> day_data = vbt.DuckDBData.pull(
...     "AAPL",
...     start="2023-10-02 09:30:00",
...     end="2023-10-02 16:00:00",
...     tz="America/New_York",
...     connection=URL
... )
>>> day_data.get()
                              Open    High     Low   Close    Volume
datetime
2023-10-02 09:30:00-04:00  171.260  171.34  170.93  171.10   61654.0
2023-10-02 09:31:00-04:00  171.130  172.37  171.13  172.30   53481.0
2023-10-02 09:32:00-04:00  172.310  172.64  172.16  172.64   44750.0
2023-10-02 09:33:00-04:00  172.640  172.97  172.54  172.78   53195.0
2023-10-02 09:34:00-04:00  172.780  173.07  172.75  173.00   47416.0
...                            ...     ...     ...     ...       ...
2023-10-02 15:55:00-04:00  173.300  173.51  173.26  173.51   61619.0
2023-10-02 15:56:00-04:00  173.525  173.59  173.42  173.43   45066.0
2023-10-02 15:57:00-04:00  173.430  173.55  173.42  173.50   45220.0
2023-10-02 15:58:00-04:00  173.510  173.60  173.46  173.56   47371.0
2023-10-02 15:59:00-04:00  173.560  173.78  173.56  173.75  161253.0

[390 rows x 5 columns]
```

## SQLAlchemy \[#sqlalchemy]

New in v2023.10.22

✅ [SQLAlchemy](https://www.sqlalchemy.org/) provides a standard interface that allows developers
to create database-agnostic code for communicating with a wide range of SQL database engines.
With its help, you can now easily store data in SQL databases and read from them as well.

```python title="Save minute data to a PostgreSQL database, and read one day"
>>> data = vbt.TVData.pull(
...     "AAPL",
...     exchange="NASDAQ",
...     timeframe="1 minute",
...     tz="America/New_York"
... )

>>> URL = "postgresql://postgres:postgres@localhost:5432"
>>> data.to_sql(engine=URL)

>>> day_data = vbt.SQLData.pull(
...     "AAPL",
...     start="2023-10-02 09:30:00",
...     end="2023-10-02 16:00:00",
...     tz="America/New_York",
...     engine=URL
... )
>>> day_data.get()
                              Open    High     Low   Close    Volume
datetime
2023-10-02 09:30:00-04:00  171.260  171.34  170.93  171.10   61654.0
2023-10-02 09:31:00-04:00  171.130  172.37  171.13  172.30   53481.0
2023-10-02 09:32:00-04:00  172.310  172.64  172.16  172.64   44750.0
2023-10-02 09:33:00-04:00  172.640  172.97  172.54  172.78   53195.0
2023-10-02 09:34:00-04:00  172.780  173.07  172.75  173.00   47416.0
...                            ...     ...     ...     ...       ...
2023-10-02 15:55:00-04:00  173.300  173.51  173.26  173.51   61619.0
2023-10-02 15:56:00-04:00  173.525  173.59  173.42  173.43   45066.0
2023-10-02 15:57:00-04:00  173.430  173.55  173.42  173.50   45220.0
2023-10-02 15:58:00-04:00  173.510  173.60  173.46  173.56   47371.0
2023-10-02 15:59:00-04:00  173.560  173.78  173.56  173.75  161253.0

[390 rows x 5 columns]
```

## PyArrow & FastParquet \[#pyarrow--fastparquet]

New in v2023.10.22

✅ Data can be written to Feather with PyArrow and to Parquet using PyArrow or FastParquet.
Parquet performs especially well in write-once, read-many scenarios, providing highly efficient data
compression and decompression, making it a great choice for storing time series data.

```python title="Save minute data to a Parquet dataset partitioned by day, and read one day"
>>> data = vbt.TVData.pull(
...     "AAPL",
...     exchange="NASDAQ",
...     timeframe="1 minute",
...     tz="America/New_York"
... )

>>> data.to_parquet(partition_by="day")  # (1)

>>> day_data = vbt.ParquetData.pull("AAPL", filters=[("group", "==", "2023-10-02")])
>>> day_data.get()
                              Open    High     Low   Close    Volume
datetime
2023-10-02 09:30:00-04:00  171.260  171.34  170.93  171.10   61654.0
2023-10-02 09:31:00-04:00  171.130  172.37  171.13  172.30   53481.0
2023-10-02 09:32:00-04:00  172.310  172.64  172.16  172.64   44750.0
2023-10-02 09:33:00-04:00  172.640  172.97  172.54  172.78   53195.0
2023-10-02 09:34:00-04:00  172.780  173.07  172.75  173.00   47416.0
...                            ...     ...     ...     ...       ...
2023-10-02 15:55:00-04:00  173.300  173.51  173.26  173.51   61619.0
2023-10-02 15:56:00-04:00  173.525  173.59  173.42  173.43   45066.0
2023-10-02 15:57:00-04:00  173.430  173.55  173.42  173.50   45220.0
2023-10-02 15:58:00-04:00  173.510  173.60  173.46  173.56   47371.0
2023-10-02 15:59:00-04:00  173.560  173.78  173.56  173.75  161253.0

[390 rows x 5 columns]
```

1.  Without `partition_by`, the data will be saved to a single Parquet file. You will still be able to
    filter rows by any column in newer versions of PyArrow and Pandas.

## Data saver \[#data-saver]

New in 1.1.1

✅ Imagine a script that can periodically pull the latest data from an exchange and save it to
disk, all without your intervention. VBT implements two classes that can do exactly this: one that
saves to CSV and another that saves to HDF.

```python title="BTCUSDT_1m_saver.py"
from vectorbtpro import *

import logging
logging.basicConfig(level=logging.INFO)

if __name__ == "__main__":
    if vbt.CSVDataSaver.file_exists():
        csv_saver = vbt.CSVDataSaver.load()
        csv_saver.update()
        init_save = False
    else:
        data = vbt.BinanceData.pull(
            "BTCUSDT",
            start="10 minutes ago UTC",
            timeframe="1 minute"
        )
        csv_saver = vbt.CSVDataSaver(data)
        init_save = True
    csv_saver.update_every(1, "minute", init_save=init_save)
    csv_saver.save()  # (1)
```

1.  The CSV data saver only stores the latest data update, which serves as the starting point for the
    next update. Be sure to save it and re-use it in the next runtime.

```console title="Run in console and then interrupt"
$ python BTCUSDT_1m_saver.py
2023-02-01 12:26:36.744000+00:00 - 2023-02-01 12:36:00+00:00: : 1it [00:01,  1.22s/it]
INFO:vectorbtpro.data.saver:Saved initial 10 rows from 2023-02-01 12:27:00+00:00 to 2023-02-01 12:36:00+00:00
INFO:vectorbtpro.utils.schedule_:Starting schedule manager with jobs [Every 1 minute do update(save_kwargs=None) (last run: [never], next run: 2023-02-01 13:37:38)]
INFO:vectorbtpro.data.saver:Saved 2 rows from 2023-02-01 12:36:00+00:00 to 2023-02-01 12:37:00+00:00
INFO:vectorbtpro.data.saver:Saved 2 rows from 2023-02-01 12:37:00+00:00 to 2023-02-01 12:38:00+00:00
INFO:vectorbtpro.utils.schedule_:Stopping schedule manager
```

```console title="Run in console again to continue"
$ python BTCUSDT_1m_saver.py
INFO:vectorbtpro.utils.schedule_:Starting schedule manager with jobs [Every 1 minute do update(save_kwargs=None) (last run: [never], next run: 2023-02-01 13:42:08)]
INFO:vectorbtpro.data.saver:Saved 5 rows from 2023-02-01 12:38:00+00:00 to 2023-02-01 12:42:00+00:00
INFO:vectorbtpro.utils.schedule_:Stopping schedule manager
```

## Local data \[#local-data]

New in 1.0.0

✅ Once you have fetched remote data, you will most likely want to save it to disk. There are two
new options for this: you can either serialize the entire data class, or save the actual data to
CSV or HDF5. Each dataset can be stored in a single flat file, which makes handling the data
easier than using a database. After saving, you can easily load the data back either by
deserializing or by using data classes that specialize in loading from CSV and HDF5 files.
These classes support a range of features, including filtering by row and datetime ranges,
updating, chunking, and even a smart dataset search that can recursively walk through
sub-directories and return datasets that match a specific glob pattern or regular expression 🧲

```python title="Fetch and save symbols separately, then load them back jointly"
>>> btc_data = vbt.BinanceData.pull("BTCUSDT")
>>> eth_data = vbt.BinanceData.pull("ETHUSDT")

>>> btc_data.to_hdf()
>>> eth_data.to_hdf()

>>> data = vbt.BinanceData.from_hdf(start="2020", end="2021")
```

Key 2/2: 100%

```python
>>> data.close
symbol                      BTCUSDT  ETHUSDT
Open time
2020-01-01 00:00:00+00:00   7200.85   130.77
2020-01-02 00:00:00+00:00   6965.71   127.19
2020-01-03 00:00:00+00:00   7344.96   134.35
2020-01-04 00:00:00+00:00   7354.11   134.20
2020-01-05 00:00:00+00:00   7358.75   135.37
...                             ...      ...
2020-12-27 00:00:00+00:00  26281.66   685.11
2020-12-28 00:00:00+00:00  27079.41   730.41
2020-12-29 00:00:00+00:00  27385.00   732.00
2020-12-30 00:00:00+00:00  28875.54   752.17
2020-12-31 00:00:00+00:00  28923.63   736.42

[366 rows x 2 columns]
```