BacktestResult.losses#

BacktestResult.losses(kind='squared', *, horizon=1, name=None, tau=None)[source]#

One loss series per origin, aligned for a comparison test.

The five losses are the squared and absolute error of the point forecast, the CRPS and log score of the predictive draws, and the pinball loss of the \(\tau\)-quantile forecast,

\[\ell_\tau(y, q) = (y - q)\bigl(\tau - \mathbb{1}\{y < q\}\bigr),\]

which is the loss a quantile forecast is optimal under. Each is returned per origin so that two backtests on the same origins can be differenced; that difference is what a ForecastComparison tests.

Parameters:

  • kind (str) – "squared" or "absolute" on the point forecast; "crps" or "log" on the predictive draws; "pinball" on the tau-quantile forecast.

  • horizon (int) – The horizon to score, in 1 .. horizons.

  • name (str | None) – A series label for a (T,) series; None returns (T, k).

  • tau (float | None) – The quantile level, required by and only by "pinball".

Return type:

NDArray[double]

Returns:

The losses, negatively oriented, (T,) or (T, k).

Raises:

SpecificationError – If the kind, horizon, name, or level is unusable, or a density loss is asked of a point backtest.

Example

>>> import numpy as np
>>> from cultivars.multivariate.reduced_form.vector_autoregression import VAR
>>> y = np.random.default_rng(0).standard_normal((130, 2))
>>> record = Backtest(y, lambda w: VAR(w, order=1).fit(), horizons=2, start=100).run()
>>> record.losses().shape, record.losses("absolute", horizon=2, name="y1").shape
((29, 2), (29,))
>>> bool(np.allclose(record.losses("squared"), record.errors[:, 0, :] ** 2))
True
>>> record.losses("crps")
Traceback (most recent call last):
    ...
cultivars.exceptions.SpecificationError: crps losses need predictive draws; ...