seqcomp: Sequential Comparison of Probabilistic Forecasts
Source:R/seqcomp-package.R
seqcomp-package.Rdseqcomp provides tools for comparing probabilistic forecasters
sequentially, following the anytime-valid framework of Choe and Ramdas
(2024).
Details
The package is built around the score difference
$$\hat{\delta}_t = S(p_t, y_t) - S(q_t, y_t),$$
where scores are positively oriented, so larger values are better. Positive
score differences favour forecaster p; negative score differences favour
forecaster q.
Main workflow
For most applications, start with compare_forecasts(). It computes
pointwise scores, running mean score differences, confidence sequences, and
e-processes in one call.
Scoring rules
The package includes positively oriented scoring rules such as
brier_score(), log_score(), spherical_score(), tick_loss(),
qlike_score(), winkler_score(), crps_normal(), crps_empirical(),
and crps_std().
Confidence sequences
Use cs_hoeffding() for Hoeffding-style confidence sequences,
cs_bernstein() for empirical Bernstein confidence sequences, and
cs_asymptotic() for asymptotic confidence sequences when finite-sample
boundedness is not available.
E-processes
Use eprocess() for the main sub-exponential mixture e-process and
eprocess_rejections() to extract first rejection times. For multi-step
forecasts, see eprocess_lag(). For predictable time-varying bounds, see
eprocess_predictable().
Winkler scores
For binary probability forecasts with unbounded base scores, use
winkler_score(), winkler_cs(), winkler_etest(), or
winkler_compare().