# Financial data pipelines (/features/data/financial-data-pipelines)

## Feature-oriented data \[#feature-oriented-data]

New in 1.14.0

✅ The main limitation of VBT's data class was that it could only store data in a
symbol-oriented format. This meant features such as OHLC had to be combined into a single DataFrame
beforehand. This approach can be somewhat counterproductive, as in VBT, we typically work with these
features separately. For example, when calling `data.close`, VBT scans for "close" columns across
all symbols, extracts them, and concatenates them into another DataFrame. To address this, the data
class has been redesigned to natively support storing feature-oriented data as well.

```python title="Create a feature-oriented data instance from various portfolio time series"
>>> data = vbt.YFData.pull(["AAPL", "MSFT", "GOOG"], tz_convert="UTC")
>>> pf = data.run("from_random_signals", n=[10, 20, 30], seed=42)

>>> pf_data = vbt.Data.from_data(
...     vbt.feature_dict({
...         "cash": pf.cash,
...         "assets": pf.assets,
...         "asset_value": pf.asset_value,
...         "value": pf.value
...     })
... )
>>> pf_data.get(feature="cash", symbol=(10, "AAPL"))
Date
1980-12-12 05:00:00+00:00      100.000000
1980-12-15 05:00:00+00:00      100.000000
1980-12-16 05:00:00+00:00      100.000000
1980-12-17 05:00:00+00:00      100.000000
1980-12-18 05:00:00+00:00      100.000000
                                      ...
2023-08-25 04:00:00+00:00     1837.648268
2023-08-28 04:00:00+00:00     1837.648268
2023-08-29 04:00:00+00:00     1837.648268
2023-08-30 04:00:00+00:00     1837.648268
2023-08-31 04:00:00+00:00     1837.648268
Name: (10, AAPL), Length: 10770, dtype: float64
```

## Parallel data \[#parallel-data]

New in 1.10.0

✅ Data fetching and updating can be easily parallelized.

```python title="Benchmark fetching multiple symbols serially and concurrently"
>>> symbols = ["SPY", "TLT", "XLF", "XLE", "XLU", "XLK", "XLB", "XLP", "XLY", "XLI", "XLV"]

>>> with vbt.Timer() as timer:
...     data = vbt.YFData.pull(symbols)
>>> print(timer.elapsed())
4.52 seconds

>>> with vbt.Timer() as timer:
...     data = vbt.YFData.pull(symbols, execute_kwargs=dict(engine="threadpool"))
>>> print(timer.elapsed())
918.54 milliseconds
```

## Runnable data \[#runnable-data]

New in 1.5.0

✅ Tired of figuring out which arguments are required by an indicator? Data instances can now
recognize the arguments of any indicator or function, map them to column names, and run the
function by passing in the required columns. You can also change the mapping, override indicator
parameters, and query indicators by name. The data instance will search all integrated indicator
packages and return the first (and best) match it finds.

```python title="Run Stochastic RSI by data"
>>> data = vbt.YFData.pull("BTC-USD")
>>> stochrsi = data.run("stochrsi")
>>> stochrsi.fastd
Date
2014-09-17 00:00:00+00:00          NaN
2014-09-18 00:00:00+00:00          NaN
2014-09-19 00:00:00+00:00          NaN
2014-09-20 00:00:00+00:00          NaN
2014-09-21 00:00:00+00:00          NaN
                                   ...
2023-01-15 00:00:00+00:00    96.168788
2023-01-16 00:00:00+00:00    91.733393
2023-01-17 00:00:00+00:00    78.295255
2023-01-18 00:00:00+00:00    48.793133
2023-01-20 00:00:00+00:00    26.242474
Name: Close, Length: 3047, dtype: float64
```

## Data parsing \[#data-parsing]

New in 1.3.0

✅ Tired of passing open, high, low, and close as separate time series? Portfolio class methods
now accept a data instance instead of just close and automatically extract the contained OHLC data.
This small but handy feature saves you time!

```python title="Backtest random signals from a data instance"
>>> data = vbt.YFData.pull("BTC-USD", start="2020-01", end="2020-03")
>>> pf = vbt.PF.from_random_signals(data, n=10, seed=42)
```

## Data transformation \[#data-transformation]

New in 1.2.3

✅ After fetching data, how do you change it? There is a new method that puts all symbols into a
single DataFrame and passes it to a UDF for transformation.

```python title="Remove weekends"
>>> data = vbt.YFData.pull(["BTC-USD", "ETH-USD"], start="2020-01-01", end="2020-01-14")
>>> new_data = data.transform(lambda df: df[~df.index.weekday.isin([5, 6])])
>>> new_data.close
symbol                         BTC-USD     ETH-USD
Date
2020-01-01 00:00:00+00:00  7200.174316  130.802002
2020-01-02 00:00:00+00:00  6985.470215  127.410179
2020-01-03 00:00:00+00:00  7344.884277  134.171707
2020-01-06 00:00:00+00:00  7769.219238  144.304153
2020-01-07 00:00:00+00:00  8163.692383  143.543991
2020-01-08 00:00:00+00:00  8079.862793  141.258133
2020-01-09 00:00:00+00:00  7879.071289  138.979202
2020-01-10 00:00:00+00:00  8166.554199  143.963776
2020-01-13 00:00:00+00:00  8144.194336  144.226593
```

## Synthetic OHLC \[#synthetic-ohlc]

New in 1.2.1

✅ New basic models are available for generating synthetic OHLC data.
These are especially useful for leakage detection.

```python title="Generate 3 months of synthetic data using Geometric Brownian Motion"
>>> data = vbt.GBMOHLCData.pull("R", start="2022-01", end="2022-04", seed=42)
>>> data.plot().show()
```

Three months of synthetic OHLC data generated with geometric Brownian motion. [Figure data (JSON)](/assets/figures/features/data/synthetic-ohlc.2cf9918db1d2.json)

## Data merging \[#data-merging]

New in 1.1.0

✅ Often, there is a need to backtest symbols from different exchanges by placing them in the
same basket. For this purpose, VBT offers a class method that can merge multiple data instances
into a single one. You can not only combine multiple symbols, but also merge datasets for a single
symbol—all done automatically!

```python title="Pull BTC datasets from various exchanges and plot the last full year relative to their mean"
>>> binance_data = vbt.CCXTData.pull("BTCUSDT", exchange="binance")
>>> bybit_data = vbt.CCXTData.pull("BTCUSDT", exchange="bybit")
>>> bitfinex_data = vbt.CCXTData.pull("BTC/USDT", exchange="bitfinex")
>>> kucoin_data = vbt.CCXTData.pull("BTC-USDT", exchange="kucoin")

>>> data = vbt.Data.merge([
...     binance_data.rename({"BTCUSDT": "Binance"}),
...     bybit_data.rename({"BTCUSDT": "Bybit"}),
...     bitfinex_data.rename({"BTC/USDT": "Bitfinex"}),
...     kucoin_data.rename({"BTC-USDT": "KuCoin"}),
... ], missing_index="drop", silence_warnings=True)

>>> @njit
... def rescale_nb(x):
...     return (x - x.mean()) / x.mean()

>>> rescaled_close = data.close.vbt.row_apply(rescale_nb)
>>> rescaled_close = rescaled_close.vbt.rolling_mean(30)
>>> last_full_year = rescaled_close.index[-1].year - 1
>>> rescaled_close.loc[str(last_full_year)].vbt.plot().show()
```

Rescaled BTC prices from Binance, Bybit, Bitfinex, and KuCoin during the last full calendar year. [Figure data (JSON)](/assets/figures/features/data/data-merging.ecc16eb2ecbd.json)