# Time-series operations (/features/tooling/time-series-operations)

## DataFrame product \[#dataframe-product]

New in 1.13.0

✅ Several parameterized indicators can produce DataFrames with different shapes and columns,
which makes creating a Cartesian product tricky because they often share common column levels
(such as "symbol") that should not be combined. There is now a method to cross-join
multiple DataFrames block-wise.

```python title="Enter when SMA goes above WMA, exit when EMA goes below WMA"
>>> data = vbt.YFData.pull(["BTC-USD", "ETH-USD"], missing_index="drop")
>>> sma = data.run("sma", timeperiod=[10, 20], unpack=True)
>>> ema = data.run("ema", timeperiod=[30, 40], unpack=True)
>>> wma = data.run("wma", timeperiod=[50, 60], unpack=True)
>>> sma, ema, wma = sma.vbt.x(ema, wma)  # (1)
>>> entries = sma.vbt.crossed_above(wma)
>>> exits = ema.vbt.crossed_below(wma)

>>> entries.columns
MultiIndex([(10, 30, 50, 'BTC-USD'),
            (10, 30, 50, 'ETH-USD'),
            (10, 30, 60, 'BTC-USD'),
            (10, 30, 60, 'ETH-USD'),
            (10, 40, 50, 'BTC-USD'),
            (10, 40, 50, 'ETH-USD'),
            (10, 40, 60, 'BTC-USD'),
            (10, 40, 60, 'ETH-USD'),
            (20, 30, 50, 'BTC-USD'),
            (20, 30, 50, 'ETH-USD'),
            (20, 30, 60, 'BTC-USD'),
            (20, 30, 60, 'ETH-USD'),
            (20, 40, 50, 'BTC-USD'),
            (20, 40, 50, 'ETH-USD'),
            (20, 40, 60, 'BTC-USD'),
            (20, 40, 60, 'ETH-USD')],
           names=['sma_timeperiod', 'ema_timeperiod', 'wma_timeperiod', 'symbol'])
```

1.  Build a Cartesian product of three DataFrames while keeping the column level "symbol" untouched.
    This can also be done with `vbt.pd_acc.cross(sma, ema, wma)`.

## Index dictionaries \[#index-dictionaries]

New in 1.3.0

Manually creating arrays and setting their data with Pandas can often be challenging. Luckily, there
is now a feature that offers much-needed assistance! Any broadcastable argument can become an index
dictionary, which contains instructions on where to set values in the array and fills them in for you.
It knows exactly which axis needs to be updated and does not create a full array unless necessary,
saving RAM ❤️

```python title="1) Accumulate daily and exit on Sunday vs 2) accumulate weekly and exit on month end"
>>> data = vbt.YFData.pull(["BTC-USD", "ETH-USD"])
>>> tile = pd.Index(["daily", "weekly"], name="strategy")  # (1)
>>> pf = vbt.PF.from_orders(
...     data.close,
...     size=vbt.index_dict({  # (2)
...         vbt.idx(
...             vbt.pointidx(every="day"),
...             vbt.colidx("daily", level="strategy")): 100,  # (3)
...         vbt.idx(
...             vbt.pointidx(every="sunday"),
...             vbt.colidx("daily", level="strategy")): -np.inf,  # (4)
...         vbt.idx(
...             vbt.pointidx(every="monday"),
...             vbt.colidx("weekly", level="strategy")): 100,
...         vbt.idx(
...             vbt.pointidx(every="monthend"),
...             vbt.colidx("weekly", level="strategy")): -np.inf,
...     }),
...     size_type="value",
...     direction="longonly",
...     init_cash="auto",
...     broadcast_kwargs=dict(tile=tile)
... )
>>> pf.sharpe_ratio
strategy  symbol
daily     BTC-USD    0.702259
          ETH-USD    0.782296
weekly    BTC-USD    0.838895
          ETH-USD    0.524215
Name: sharpe_ratio, dtype: float64
```

1.  To represent two strategies, you need to tile the same data twice. Create a parameter with
    strategy names and pass it as `tile` to the broadcaster so it tiles the columns of each array
    (such as price) twice.
2.  The index dictionary includes index instructions as keys and data as values to set.
    Keys can be row indices, labels, or custom indexer classes such as `PointIdxr`.
3.  Find the indices of the rows for the start of each day and the column index of "daily", then
    set each element at those indices to 100 (= accumulate).
4.  Find the indices of the rows that correspond to Sunday. If any value at those indices has already
    been set by a previous instruction, it will be overridden.

## Slicing \[#slicing]

New in 1.3.0

✅ Similar to selecting columns, each VBT object can now slice rows using the
same mechanism as in Pandas 🔪 This makes it easy to analyze and plot any subset of
simulated data, without needing to re-simulate!

```python title="Analyze multiple date ranges of the same portfolio"
>>> data = vbt.YFData.pull("BTC-USD")
>>> pf = vbt.PF.from_holding(data, freq="d")

>>> pf.sharpe_ratio
1.116727709477293

>>> pf.loc[:"2020"].sharpe_ratio  # (1)
1.2699801554196481

>>> pf.loc["2021": "2021"].sharpe_ratio  # (2)
0.9825161170278687

>>> pf.loc["2022":].sharpe_ratio  # (3)
-1.0423271337174647
```

1.  Get the Sharpe ratio during the year 2020 and before.
2.  Get the Sharpe ratio during the year 2021.
3.  Get the Sharpe ratio during the year 2022 and after.

## Column stacking \[#column-stacking]

New in 1.3.0

✅ Complex VBT objects of the same type can be easily stacked along columns. For example,
you can combine multiple unrelated trading strategies into one portfolio for analysis.
Under the hood, the final object is still represented as a monolithic multi-dimensional structure
that can be processed even faster than separate merged objects 🫁

```python title="Analyze two trading strategies separately and then jointly"
>>> def strategy1(data):
...     fast_ma = vbt.MA.run(data.close, 50, short_name="fast_ma")
...     slow_ma = vbt.MA.run(data.close, 200, short_name="slow_ma")
...     entries = fast_ma.ma_crossed_above(slow_ma)
...     exits = fast_ma.ma_crossed_below(slow_ma)
...     return vbt.PF.from_signals(
...         data.close,
...         entries,
...         exits,
...         size=100,
...         size_type="value",
...         init_cash="auto"
...     )

>>> def strategy2(data):
...     bbands = vbt.BBANDS.run(data.close, window=14)
...     entries = bbands.close_crossed_below(bbands.lower)
...     exits = bbands.close_crossed_above(bbands.upper)
...     return vbt.PF.from_signals(
...         data.close,
...         entries,
...         exits,
...         init_cash=200
...     )

>>> data1 = vbt.BinanceData.pull("BTCUSDT")
>>> pf1 = strategy1(data1)  # (1)
>>> pf1.sharpe_ratio
0.9100317671866922

>>> data2 = vbt.BinanceData.pull("ETHUSDT")
>>> pf2 = strategy2(data2)  # (2)
>>> pf2.sharpe_ratio
-0.11596286232734827

>>> pf_sep = vbt.PF.column_stack((pf1, pf2))  # (3)
>>> pf_sep.sharpe_ratio
0    0.910032
1   -0.115963
Name: sharpe_ratio, dtype: float64

>>> pf_join = vbt.PF.column_stack((pf1, pf2), group_by=True)  # (4)
>>> pf_join.sharpe_ratio
0.42820898354646514
```

1.  Analyze the first strategy in its own portfolio.
2.  Analyze the second strategy in its own portfolio.
3.  Analyze both strategies separately in the same portfolio.
4.  Analyze both strategies jointly in the same portfolio.

## Row stacking \[#row-stacking]

New in 1.3.0

✅ Complex VBT objects of the same type can be easily stacked along rows. For example,
you can append new data to an existing portfolio, or concatenate in-sample portfolios with
their out-of-sample counterparts 🧬

```python title="Analyze two date ranges separately and then jointly"
>>> def strategy(data, start=None, end=None):
...     fast_ma = vbt.MA.run(data.close, 50, short_name="fast_ma")
...     slow_ma = vbt.MA.run(data.close, 200, short_name="slow_ma")
...     entries = fast_ma.ma_crossed_above(slow_ma)
...     exits = fast_ma.ma_crossed_below(slow_ma)
...     return vbt.PF.from_signals(
...         data.close[start:end],
...         entries[start:end],
...         exits[start:end],
...         size=100,
...         size_type="value",
...         init_cash="auto"
...     )

>>> data = vbt.BinanceData.pull("BTCUSDT")

>>> pf_whole = strategy(data)  # (1)
>>> pf_whole.sharpe_ratio
0.9100317671866922

>>> pf_sub1 = strategy(data, end="2019-12-31")  # (2)
>>> pf_sub1.sharpe_ratio
0.7810397448678937

>>> pf_sub2 = strategy(data, start="2020-01-01")  # (3)
>>> pf_sub2.sharpe_ratio
1.070339534746574

>>> pf_join = vbt.PF.row_stack((pf_sub1, pf_sub2))  # (4)
>>> pf_join.sharpe_ratio
0.9100317671866922
```

1.  Analyze the entire range.
2.  Analyze the first date range.
3.  Analyze the second date range.
4.  Combine both date ranges and analyze them together.

## Index alignment \[#index-alignment]

New in 1.3.0

✅ There is no longer a limitation requiring each Pandas array to have the same index.
Indexes of all arrays that should broadcast against each other are automatically aligned, as long as they
have the same data type.

```python title="Predict ETH price with BTC price using linear regression"
>>> btc_data = vbt.YFData.pull("BTC-USD")
>>> btc_data.wrapper.shape
(2817, 7)

>>> eth_data = vbt.YFData.pull("ETH-USD")  # (1)
>>> eth_data.wrapper.shape
(1668, 7)

>>> ols = vbt.OLS.run(  # (2)
...     btc_data.close,
...     eth_data.close
... )
>>> ols.pred
Date
2014-09-17 00:00:00+00:00            NaN
2014-09-18 00:00:00+00:00            NaN
2014-09-19 00:00:00+00:00            NaN
2014-09-20 00:00:00+00:00            NaN
2014-09-21 00:00:00+00:00            NaN
...                                  ...
2022-05-30 00:00:00+00:00    2109.769242
2022-05-31 00:00:00+00:00    2028.856767
2022-06-01 00:00:00+00:00    1911.555689
2022-06-02 00:00:00+00:00    1930.169725
2022-06-03 00:00:00+00:00    1882.573170
Freq: D, Name: Close, Length: 2817, dtype: float64
```

1.  ETH-USD history is shorter than BTC-USD history.
2.  This now works! Make sure all arrays share the same timeframe and timezone.

## Numba datetime \[#numba-datetime]

New in 1.2.3

✅ Numba does not support datetime indexes (or any other Pandas objects). There are
also no built-in Numba functions for working with datetime. So, how do you connect data to time? VBT
addresses this gap by implementing a collection of functions to extract various information
from each timestamp, such as the current time and day of the week, to determine whether the bar
is during trading hours.

```python title="Plot the percentage change from the start of the month to now"
>>> @njit
... def month_start_pct_change_nb(arr, index):
...     out = np.full(arr.shape, np.nan)
...     for col in range(arr.shape[1]):
...         for i in range(arr.shape[0]):
...             if i == 0 or vbt.dt_nb.month_nb(index[i - 1]) != vbt.dt_nb.month_nb(index[i]):
...                 month_start_value = arr[i, col]
...             else:
...                 out[i, col] = (arr[i, col] - month_start_value) / month_start_value
...     return out

>>> data = vbt.YFData.pull(["BTC-USD", "ETH-USD"], start="2022", end="2023")
>>> pct_change = month_start_pct_change_nb(
...     vbt.to_2d_array(data.close),
...     data.index.vbt.to_ns()  # (1)
... )
>>> pct_change = data.symbol_wrapper.wrap(pct_change)
>>> pct_change.vbt.plot().show()
```

1.  Convert the datetime index to nanosecond format.

Bitcoin and Ethereum percentage change from the start of each month in 2022. [Figure data (JSON)](/assets/figures/features/productivity/numba-datetime.b4837f52d042.json)

!!! info "Tutorial"
    Learn more in the [Signal development](/tutorials/signal-development) tutorial.

## Periods ago \[#periods-ago]

New in 1.2.3

✅ Instead of writing Numba functions, comparing values at different bars can also be done in a vectorized
way with Pandas. The problem is that there are no built-in functions to easily shift values based on
timedeltas, nor are there rolling functions to check whether an event happened during a past
period. This gap is filled by various new accessor methods.

```python title="Check whether the price dropped for 5 consecutive bars"
>>> data = vbt.YFData.pull("BTC-USD", start="2022-05", end="2022-08")
>>> mask = (data.close < data.close.vbt.ago(1)).vbt.all_ago(5)
>>> fig = data.plot(plot_volume=False)
>>> mask.vbt.signals.ranges.plot_shapes(
...     plot_close=False,
...     fig=fig,
...     add_shape_kwargs=dict(fillcolor="orangered")
... )
>>> fig.show()
```

Bitcoin price with highlighted ranges where price dropped for five consecutive bars. [Figure data (JSON)](/assets/figures/features/productivity/periods-ago.ce2f9924664f.json)

!!! info "Tutorial"
    Learn more in the [Signal development](/tutorials/signal-development) tutorial.

## Safe resampling \[#safe-resampling]

New in 1.1.2

✅ [Look-ahead bias](https://www.investopedia.com/terms/l/lookaheadbias.asp) is an ongoing
risk when working with array data, especially on multiple time frames. Using Pandas alone is strongly
discouraged because it does not recognize that financial data mainly involves bars where timestamps are
the opening times, and events may occur at any time between bars. Pandas thus incorrectly assumes that
timestamps indicate the exact time of an event. In VBT, there is a complete collection of functions and
classes for safely resampling and analyzing data!

```python title="Calculate SMA on multiple time frames and display on the same chart"
>>> def mtf_sma(close, close_freq, target_freq, timeperiod=5):
...     target_close = close.vbt.realign_closing(target_freq)  # (1)
...     target_sma = vbt.talib("SMA").run(target_close, timeperiod=timeperiod).real  # (2)
...     target_sma = target_sma.rename(f"SMA ({target_freq})")
...     return target_sma.vbt.realign_closing(close.index, freq=close_freq)  # (3)

>>> data = vbt.YFData.pull("BTC-USD", start="2020", end="2023")
>>> fig = mtf_sma(data.close, "D", "daily").vbt.plot()
>>> mtf_sma(data.close, "D", "weekly").vbt.plot(fig=fig)
>>> mtf_sma(data.close, "D", "monthly").vbt.plot(fig=fig)
>>> fig.show()
```

1.  Resample the source frequency to the target frequency. Since Close occurs at the end of the bar,
    resample it as a "closing event".
2.  Calculate the SMA on the target frequency.
3.  Resample the target frequency back to the source frequency to show
    multiple time frames on the same chart. Because `close` contains gaps, you cannot simply resample to `close_freq`
    as this might produce unaligned series. Instead, resample directly to the index of `close`.

Bitcoin simple moving averages calculated on daily, weekly, and monthly timeframes. [Figure data (JSON)](/assets/figures/features/productivity/safe-resampling.5bdd558681a0.json)

!!! info "Tutorial"
    Learn more in the [MTF analysis](/tutorials/mtf-analysis) tutorial.

## Resamplable objects \[#resamplable-objects]

New in 1.1.2

✅ You can resample not only time series, but also complex VBT objects! Under the hood,
each object is made up of a collection of array-like attributes, so resampling means aggregating
all the related information together. This is especially helpful if you want to simulate at a higher
frequency for maximum accuracy and then analyze at a lower frequency for better speed.

```python title="Plot the monthly return heatmap of a random portfolio"
>>> import calendar

>>> data = vbt.YFData.pull("BTC-USD", start="2018", end="2023")
>>> pf = vbt.PF.from_random_signals(data, n=100, direction="both", seed=42)
>>> mo_returns = pf.resample("M").returns  # (1)
>>> mo_return_matrix = pd.Series(
...     mo_returns.values,
...     index=pd.MultiIndex.from_arrays([
...         mo_returns.index.year,
...         mo_returns.index.month
...     ], names=["year", "month"])
... ).unstack("month")
>>> mo_return_matrix.columns = mo_return_matrix.columns.map(lambda x: calendar.month_abbr[x])
>>> mo_return_matrix.vbt.heatmap(
...     is_x_category=True,
...     trace_kwargs=dict(zmid=0, colorscale="Spectral")
... ).show()
```

1.  Resample the entire portfolio to monthly frequency and calculate the returns.

Monthly return heatmap for a seeded random Bitcoin portfolio from 2018 through 2022. [Figure data (JSON)](/assets/figures/features/productivity/monthly-return-heatmap.36c65fb0009f.json)

!!! info "Tutorial"
    Learn more in the [MTF analysis](/tutorials/mtf-analysis) tutorial.

## Meta methods \[#meta-methods]

New in 1.0.0

✅ Many methods, such as rolling apply, now come in two versions: regular (instance methods)
and meta (class methods). Regular methods are bound to a single array and do not need metadata,
while meta methods are not tied to any array and act as micro-pipelines with their own
broadcasting and templating logic. Here, VBT solves one of the main Pandas limitations:
the inability to apply a function to multiple arrays at once.

```python title="Compute the rolling z-score on one array and the rolling correlation coefficient on two arrays"
>>> @njit
... def zscore_nb(x):  # (1)
...     return (x[-1] - np.mean(x)) / np.std(x)

>>> data = vbt.YFData.pull("BTC-USD", start="2020", end="2021")
>>> data.close.rolling(14).apply(zscore_nb, raw=True)  # (2)
Date
2020-01-01 00:00:00+00:00         NaN
                                  ...
2020-12-27 00:00:00+00:00    1.543527
2020-12-28 00:00:00+00:00    1.734715
2020-12-29 00:00:00+00:00    1.755125
2020-12-30 00:00:00+00:00    2.107147
2020-12-31 00:00:00+00:00    1.781800
Freq: D, Name: Close, Length: 366, dtype: float64

>>> data.close.vbt.rolling_apply(14, zscore_nb)  # (3)
2020-01-01 00:00:00+00:00         NaN
                                  ...
2020-12-27 00:00:00+00:00    1.543527
2020-12-28 00:00:00+00:00    1.734715
2020-12-29 00:00:00+00:00    1.755125
2020-12-30 00:00:00+00:00    2.107147
2020-12-31 00:00:00+00:00    1.781800
Freq: D, Name: Close, Length: 366, dtype: float64

>>> @njit
... def corr_meta_nb(from_i, to_i, col, a, b):  # (4)
...     a_window = a[from_i:to_i, col]
...     b_window = b[from_i:to_i, col]
...     return np.corrcoef(a_window, b_window)[1, 0]

>>> data2 = vbt.YFData.pull(["ETH-USD", "XRP-USD"], start="2020", end="2021")
>>> vbt.pd_acc.rolling_apply(  # (5)
...     14,
...     corr_meta_nb,
...     vbt.Rep("a"),
...     vbt.Rep("b"),
...     broadcast_named_args=dict(a=data.close, b=data2.close)
... )
symbol                      ETH-USD   XRP-USD
Date
2020-01-01 00:00:00+00:00       NaN       NaN
...                             ...       ...
2020-12-27 00:00:00+00:00  0.636862 -0.511303
2020-12-28 00:00:00+00:00  0.674514 -0.622894
2020-12-29 00:00:00+00:00  0.712531 -0.773791
2020-12-30 00:00:00+00:00  0.839355 -0.772295
2020-12-31 00:00:00+00:00  0.878897 -0.764446

[366 rows x 2 columns]
```

1.  Provides access to the window only.
2.  Using Pandas.
3.  Using the regular method, which accepts the same function as Pandas.
4.  Provides access to one or more entire arrays.
5.  Using the meta method, which accepts metadata and variable arguments.

## Array expressions \[#array-expressions]

New in 1.0.0

✅ When combining multiple arrays, they often need to be aligned and broadcast before
the operation itself. Pandas alone often falls short because it can be too strict.
Fortunately, VBT includes an accessor class method that can take a regular Python expression,
identify all variable names, extract the arrays from the current context, broadcast them,
and then evaluate the expression (with support for [NumExpr](https://github.com/pydata/numexpr)!) ⌨️

```python title="Evaluate a multiline array expression based on a Bollinger Bands indicator"
>>> data = vbt.YFData.pull(["BTC-USD", "ETH-USD"])

>>> low = data.low
>>> high = data.high
>>> bb = vbt.talib("BBANDS").run(data.close)
>>> upperband = bb.upperband
>>> lowerband = bb.lowerband
>>> bandwidth = (bb.upperband - bb.lowerband) / bb.middleband
>>> up_th = vbt.Param([0.3, 0.4])
>>> low_th = vbt.Param([0.1, 0.2])

>>> expr = """
... narrow_bands = bandwidth < low_th
... above_upperband = high > upperband
... wide_bands = bandwidth > up_th
... below_lowerband = low < lowerband
... (narrow_bands & above_upperband) | (wide_bands & below_lowerband)
... """
>>> mask = vbt.pd_acc.eval(expr)
>>> mask.sum()
low_th  up_th  symbol
0.1     0.3    BTC-USD    344
               ETH-USD    171
        0.4    BTC-USD    334
               ETH-USD    158
0.2     0.3    BTC-USD    444
               ETH-USD    253
        0.4    BTC-USD    434
               ETH-USD    240
dtype: int64
```