The benchmark forecasts every model must beat

Today let's understand the benchmark forecasts every model must beat, using one running example you'll carry through the whole lesson.

Dockside Kayak Rentals is a small rental shop. Its owner keeps one simple record: how many kayaks went out the door each month, from January 2022 to December 2025. That's 48 months.

Look at the shape above. Rentals climb year over year, and every summer they spike well above winter's low point, then fall back each January. Before reaching for anything more complicated, there's a simpler question worth answering first: what's the plainest forecast you could make for a series shaped like this, and how good would it have to be before a fancier model is worth switching to?

Splitting the series into what you fit and what you check

Before fitting any model, you need a plain, boring rule to compare it against. That's called a benchmark forecast: a rule simple enough to need almost no judgment to build, yet it sets the minimum bar any fancier model has to clear. If a complicated model can't beat a benchmark, its extra complexity wasn't worth it.

To check a forecast fairly, you can't look at how well it fits months it already saw. You have to hold real months back, forecast them without looking, and then compare. So split Dockside Kayak Rentals' 48 months into two windows: a training window the models get to see, and a test window they don't.

Thirty-six months, 2022 through 2024, train every benchmark this lesson builds. The twelve months of 2025 stay locked away as test, the real numbers every forecast gets checked against.

Now plot the whole series again, this time colouring the two windows so the split is easy to see.

Blue is train, the history every model gets to learn from. Orange is test, the twelve months no model is allowed to see until after it's made its forecast.

The plainest guesses: MEAN() and NAIVE()

Start with the two simplest forecasts anyone could make.

The first is MEAN(): forecast every future month as the historical average, the mean of everything you've seen so far. The second is NAIVE(): forecast every future month as whatever the very last observed value was, and just repeat it forward. Neither one looks at trend or season. Both just draw a flat line.

Fit both on train, using fable's model() function, and forecast 12 months ahead, the length of the held-back test window.

Mean forecasts a flat 181.2 rentals every month of 2025, the average of all 36 training months. Naive forecasts a flat 156.0, because December 2024, the last month train saw, happened to land at 156. Neither number moves from January to December, even though the real series climbs into summer and dips into winter every single year.

Plot both against the training history to see the mismatch directly.

Notice Naive's flat line sits low, because it anchors on a winter month, and that same low number carries straight through a summer test window that historically runs far above 156. Mean at least sits in the middle of the range, but it still can't rise for summer or fall for winter, because a flat line by definition can't do either.

Repeating what happened last year: SNAIVE()

MEAN() and NAIVE() both missed the season entirely. The next benchmark is built to catch it.

SNAIVE(), short for seasonal naive, forecasts a month as whatever that same month did one year earlier. Its forecast for July 2025 is just July 2024's actual value, carried forward unchanged. To do that, SNAIVE() needs to know the length of one season, which fable reads straight off the tsibble's monthly index as 12, so it always looks back exactly 12 rows.

Every one of those twelve numbers is a 2024 value, copied straight across: 125 in January, climbing to 281 in July, falling back to 156 in December. Seasonal_naive never learns a trend and never learns a level. It just assumes this year repeats last year's shape, month for month.

Here's the real 2025 test window, the twelve months Seasonal_naive is trying to guess.

Compare that shape with the forecast values in the table above. The rise into summer and the fall into winter are both there in the real 2025 data, in the same places Seasonal_naive predicted them, because it copied the shape straight from 2024. For a series with a real repeating season, copying last year's shape alone already puts Seasonal_naive far ahead of Mean and Naive.

Carrying the trend forward: RW(y ~ drift())

Seasonal_naive gets the shape right. But the real 2025 numbers don't just repeat 2024, they run a bit higher too: July 2024 saw 281 rentals, and July 2025 saw 292. Rentals are still climbing year over year, on top of the season. The last benchmark is built to catch that kind of rising or falling level, not a repeating season.

It's called the drift model, written RW(y ~ drift()) in fable. It works like NAIVE(), the last observed value, plus one constant step added at every future month. That step is the average change per month across the whole training window: the last training value minus the first, divided by the number of steps between them.

January 2022 opened at 82 rentals and December 2024 closed at 156, a rise of 74 rentals spread over 35 monthly steps, or about 2.11 rentals a month. RW(rentals ~ drift())'s forecast is exactly NAIVE()'s 156 plus that step, added again and again: 158.1 in January, climbing steadily to 181.4 by December. It's a straight sloped line, the one shape Mean, Naive and Seasonal_naive none of them draw.

Drift's straight line does catch the general climb, but it can't bend up for summer or down for winter the way Seasonal_naive does. For a series with real seasonality like this one, carrying the trend forward isn't enough on its own. It's missing the one thing Seasonal_naive got right.

Quick check: which benchmark fits which series

Suppose you're handed a new series: it swings up and down in a clear, repeating pattern every year, but shows no real long-term climb or fall. Which benchmark would you reach for first?

Scoring every forecast against the real 2025 numbers

So far you've judged each benchmark by eye, comparing shapes on a chart. Now put a real number on each one.

Fit all four benchmarks together, forecast 2025, and score every forecast against the real values in kayak using fable's accuracy() function.

RMSE (root mean squared error) and MAE (mean absolute error) both measure how far off a forecast typically was, in rentals per month, so smaller is better for both. Seasonal_naive comes out lowest on both, 24.8 and 21.3, matching what you already saw on the chart: it's the only benchmark that traced the real shape. Naive comes out worst, 85.5 and 70.8, because that one flat line anchored on a winter trough missed the entire summer. Mean and Drift land in between, and Mean edges out Drift, 67.6 against 74.0, because sitting in the middle of the range beats a straight climb that still can't bend for the season.

MASE: scoring against the benchmark itself

RMSE and MAE tell you how far off each forecast was, in rentals. But rentals is just this series' own unit. A model scoring RMSE 25 would be excellent for a shop this size and terrible for a shop renting thousands a month. You need a score that means the same thing regardless of scale, and that's what MASE gives you.

MASE stands for mean absolute scaled error. It's a model's mean absolute error on the test window, divided by one specific number: the mean absolute error NAIVE() would have made stepping through the training data itself, one month at a time. That denominator is not the same thing as how NAIVE() itself scored on the real 2025 test months, RMSE 85.5 and MAE 70.8.

It's a separate number, computed by walking through the 36 training months and asking, for each one, how far off would a naive guess (last month's value) have been. Divide by that, and a MASE under 1 means a model beat what a naive guess would have scored on training data alone. A MASE at or above 1 means it didn't.

Seasonal_naive's MASE comes out at 0.87, comfortably under 1. Mean, Drift and Naive all land above 1, with Naive the worst of the three at 2.90. So Seasonal_naive isn't just the best of these four benchmarks on RMSE and MAE, it's the only one that beats a plain naive guess by this scaled measure at all.

Your turn: beat SNAIVE with a fifth model

Seasonal_naive repeats last year's shape but never accounts for the fact that rentals are still climbing year over year. fable lets you combine both ideas in one model: add a drift() term to SNAIVE()'s own formula, the same way RW(rentals ~ drift())'s formula adds one to a plain random walk.

Fill in the blank below to add a fifth model, Seasonal_naive_drift, to the fitted set, then rerun accuracy() to see if it beats Seasonal_naive.

# Seasonal naive plus a drift term, refit and rescored against the other four
fit_all2 <- model(train,
  Mean = MEAN(rentals),
  Naive = NAIVE(rentals),
  Seasonal_naive = SNAIVE(rentals),
  Drift = RW(rentals ~ drift()),
  Seasonal_naive_drift = SNAIVE(rentals ~ drift())
)

fc_all2 <- forecast(fit_all2, h = 12)
acc_all2 <- accuracy(fc_all2, kayak)

print(as.data.frame(acc_all2 |> as_tibble() |> transmute(model = .model, RMSE = round(RMSE, 1), MAE = round(MAE, 1), MASE = round(MASE, 2)) |> arrange(MASE)), row.names = FALSE)
#>                 model RMSE  MAE MASE
#>  Seasonal_naive_drift 12.7 11.6 0.48
#>        Seasonal_naive 24.8 21.3 0.87
#>                  Mean 67.6 57.6 2.36
#>                 Drift 74.0 61.2 2.51
#>                 Naive 85.5 70.8 2.90

Quick check: what beating a benchmark really means

Picture a different series, a different fancier model. On its withheld test window, that model scores MASE 1.15, while Seasonal_naive scores 0.82 on the same window. What should you do?

References

• Forecasting: Principles and Practice, section 5.2, Some simple forecasting methods - Hyndman and Athanasopoulos (3rd ed.), the standard reference for MEAN(), NAIVE(), SNAIVE() and the drift method.
• Forecasting: Principles and Practice, section 5.8, Evaluating point forecast accuracy - Hyndman and Athanasopoulos (3rd ed.), the source for RMSE, MAE and MASE as used in this lesson.
• Hyndman, R.J. and Koehler, A.B. (2006), "Another look at measures of forecast accuracy," International Journal of Forecasting, 22(4), 679-688. The paper that introduced MASE.
• fable package reference documentation for MEAN(), NAIVE(), SNAIVE() and RW()
• fabletools package reference documentation for accuracy() and its measures

What you can do now

You can now fit all four benchmark forecasts on a tsibble and explain what each one draws: MEAN() a flat line at the historical average, NAIVE() a flat line at the last observed value, SNAIVE() last year's shape repeated, and RW(y ~ drift()) a straight sloped line carrying the average change forward.

You can score any set of fitted models with accuracy() against real withheld data, read RMSE and MAE in the series' own units, and read MASE as a scale-free score against a naive guess made inside the training data itself, under 1 to beat it, at or above 1 to lose to it.

And you know the rule that ties it together: added complexity is only worth it once a model beats the best benchmark on data it never saw during fitting. Looking more sophisticated is not the same thing as forecasting better.

From here, every forecasting model built on this series gets judged against this same benchmark score.