Skip to contents

Compare Multiple Sequential Forecasters (SMCS)

Source: R/smcs_compare.R
smcs_compare.Rd

Evaluates an arbitrary number of candidate forecasters simultaneously, constructing Sequential Model Confidence Sets (SMCS) that maintain family-wise error rate control over time.

Usage

smcs_compare(
  forecasts,
  outcomes,
  scoring_rule = c("brier", "spherical", "tick"),
  cs_method = c("bernstein", "hoeffding"),
  tau = NULL,
  alpha = 0.05,
  v_opt = 10,
  clip_max = 1e+07,
  betting_rule = c("default", "naive", "agrapa", "ons", "arnold"),
  period = 1,
  ...
)

Arguments

forecasts

A \(T \times m\) matrix of forecasts. For binary/categorical probability forecasts, these should be matrices of probabilities. For quantile forecasts, these should be raw predicted quantiles. Note: to replicate the log-scale bounds of the Arnold et al. (2026) Covid-19 study, pass log-transformed forecasts and outcomes.

outcomes

A numeric vector of \(T\) realised outcomes.

scoring_rule

Character. Scoring rule used to compare forecasts. Currently supports "brier", "spherical", and "tick".

cs_method

Character. Confidence sequence method for the weak null: "bernstein" or "hoeffding". Default is "bernstein".

tau

Numeric in (0, 1). The quantile level. Required only if scoring_rule = "tick".

alpha

Numeric in (0, 1). Family-wise significance level. Default is 0.05.

v_opt

Numeric > 0. Intrinsic time at which the weak-null confidence sequence is tuned to be tightest. Default is 10.

clip_max

Numeric. Maximum e-process value before clipping in the strong-null test. Default is 1e7.

betting_rule

Character. Strong-null betting-fraction rule. "default" preserves the existing behavior: "naive" for "brier" and "spherical", and "arnold" for "tick". "naive", "agrapa", and "ons" are available for all scoring rules; "arnold" is available only for "tick".

period

Positive integer. Number of interleaved periodic sub-streams used by "agrapa" or "ons". Must be 1 for "naive" and "arnold".

...

Additional arguments passed to build_agrapa_betting_array() or build_ons_betting_array() for the selected adaptive rule, such as kappa, prior_mean, prior_variance, fake_obs, or eta.

Value

A list containing:

scores

A \(T \times m\) matrix of evaluated pointwise scores.

smcs_strong

A \(T \times m\) logical matrix tracking inclusion in the strong-null SMCS over time (permanent exclusions).

smcs_uniform_weak

A \(T \times m\) logical matrix tracking inclusion in the uniformly weak SMCS over time (permanent exclusions). Currently NULL for "tick" loss.

smcs_weak

A \(T \times m\) logical matrix tracking inclusion in the time-varying weak-null SMCS over time (models can exit and re-enter). Currently NULL for "tick" loss.

betting_rule

The resolved strong-null betting rule actually used.

period

The period supplied for an adaptive betting rule.

betting_args

The additional adaptive-rule settings supplied through ....

Details

This is a high-level wrapper that automates pointwise score calculation, boundary generation, and multiplicity corrections via smcs_strong() and smcs_weak(). For uniformly bounded scoring rules, it returns SMCSs under the strong, uniformly weak, and time-varying weak null hypotheses. For conditionally bounded rules like "tick" loss, it automatically builds the dynamic 3D arrays required for strong-null adaptive betting.

Examples

set.seed(42)
T_sim <- 100
y <- rbinom(T_sim, 1, 0.5)

# Create 3 forecasters:
# M1 is a perfect oracle (always predicts the true y)
# M2 is slightly noisy (adds small uniform noise to y)
# M3 is an anti-oracle (predicts the exact opposite of y)
fcsts <- matrix(NA, nrow = T_sim, ncol = 3)
fcsts[, 1] <- y
fcsts[, 2] <- abs(y - runif(T_sim, 0, 0.1))
fcsts[, 3] <- 1 - y
colnames(fcsts) <- c("M1", "M2", "M3")

out <- smcs_compare(fcsts, y, scoring_rule = "brier")

# Print the object to see exclusions (M3 will be dropped rapidly)
out
#> <seqcomp multi-model comparison>
#>   3 models, 100 time steps, alpha = 0.05, scoring rule = 'brier', strong betting rule = 'naive'
#> 
#> -- Strong-null SMCS (Permanent Exclusion) --
#>  model status_at_T first_dropped
#>     M1    included             -
#>     M2    included             -
#>     M3    excluded            19
#>    final set size: 3 -> 2
#> 
#> -- Uniformly-weak SMCS (Permanent Exclusion) --
#>  model status_at_T first_dropped
#>     M1    included             -
#>     M2    included             -
#>     M3    excluded            11
#>    final set size: 3 -> 2
#> 
#> -- Weak-null SMCS (Dynamic Re-entry) --
#>  model status_at_T first_dropped
#>     M1    included             -
#>     M2    included             -
#>     M3    excluded            15
#>    final set size: 3 -> 2
#> 
#> Use `x$smcs_strong`, `x$smcs_uniform_weak`, or `x$smcs_weak` for full  inclusion matrices, or `x$scores` for pointwise scores.

# View how the set sizes shrink over time
summary(out)
#>     t strong_set_size uniform_weak_set_size weak_set_size
#> 1   1               3                     3             3
#> 2  25               2                     2             2
#> 3  50               2                     2             2
#> 4  75               2                     2             2
#> 5 100               2                     2             2

# Additionally, use an adaptive strong-null betting rule
out_agrapa <- smcs_compare(
  fcsts, y, scoring_rule = "brier", betting_rule = "agrapa"
)

out_agrapa
#> <seqcomp multi-model comparison>
#>   3 models, 100 time steps, alpha = 0.05, scoring rule = 'brier', strong betting rule = 'agrapa'
#> 
#> -- Strong-null SMCS (Permanent Exclusion) --
#>  model status_at_T first_dropped
#>     M1    included             -
#>     M2    included             -
#>     M3    excluded            12
#>    final set size: 3 -> 2
#> 
#> -- Uniformly-weak SMCS (Permanent Exclusion) --
#>  model status_at_T first_dropped
#>     M1    included             -
#>     M2    included             -
#>     M3    excluded            11
#>    final set size: 3 -> 2
#> 
#> -- Weak-null SMCS (Dynamic Re-entry) --
#>  model status_at_T first_dropped
#>     M1    included             -
#>     M2    included             -
#>     M3    excluded            15
#>    final set size: 3 -> 2
#> 
#> Use `x$smcs_strong`, `x$smcs_uniform_weak`, or `x$smcs_weak` for full  inclusion matrices, or `x$scores` for pointwise scores.