Criteria for a good forecasting method

Summary

A good forecasting method fits the decision that follows the forecast. It uses information that will be available at prediction time, produces the required horizon and level of detail, improves on a relevant baseline, and can run within the operating constraints of the business. Accuracy matters. Bias, stability, response time, cost, explainability, failure handling, and planner use also matter. Evaluate the complete forecasting path before selecting a method.

Define the decision first

Record the operating decision before comparing algorithms. A replenishment forecast supports a different decision from a financial forecast or a workforce forecast. Define these fields: The decision definition creates the acceptance test. A method that performs well at a monthly category level can still fail an item-level replenishment workflow. Write the definition where the planning team can see it, and revisit it when the decision changes. A forecast built for weekly replenishment will quietly fail when the business starts using it for a monthly financial commitment.

Forecast target, such as units, orders, revenue, visits, or workload
Decision grain, such as item and store by day
Forecast horizon and update cadence
Lead time between forecast and action
Person or system that acts on the result
Cost of forecasting above or below actual demand
Required service level and review time

Check data availability at the forecast cutoff

Training data must reproduce what the system will know when it creates a forecast. Record event time, ingestion time, correction time, and the date when each future-known input became available. Review these data conditions: A backtest contains leakage when it uses information that arrived after the simulated cutoff. Leakage can make an unsuitable model appear strong. Price and promotion calendars deserve special care. They are often known in advance, which makes them future-known inputs, but late changes to them are common. Record both the plan and its correction history.

Length and continuity of history
Seasonality and calendar coverage
Price, promotion, event, and weather availability
Stockouts, lost sales, returns, cancellations, and transfers
Product launches, replacements, and retirements
Location openings, closures, and operating-hour changes
Late corrections and changing source definitions

Use a baseline that represents current performance

Compare every candidate with a method that the team understands. Useful baselines include the last value, the same period last week, a seasonal naive forecast, a moving average, and the forecast planners use today. The candidate should earn the added data, infrastructure, review, and operating cost. Keep the strongest safe baseline available as a fallback after release. Publish baseline results with every candidate evaluation. A candidate that beats a naive forecast by a small margin and costs a team to maintain has a weaker case than its accuracy chart suggests.

Match the method to the demand pattern

Stable seasonal demand may need a seasonal baseline or exponential smoothing. Autocorrelated series with enough history may suit autoregressive integrated moving average (ARIMA) methods. Intermittent demand requires methods and evaluation that account for long zero-demand periods. Regression and tree-based methods become useful when reliable drivers add information. Global or deep models can help across a large related portfolio when the data and operating team can support them. Segment the portfolio by volume, variability, intermittency, history length, promotion sensitivity, product life cycle, margin, or service importance. A routing policy can assign different methods to different segments.

Run a structured selection trial

Treat method selection as a controlled experiment, not a benchmark download. Give the trial a time limit and a decision date. A selection process without an end date tends to add candidates instead of choosing a method the team can run.

Select representative series: high volume, intermittent, seasonal, promoted, new, and declining.
Freeze the cutoff dates and the information available at each one.
Run the baselines first and record their results.
Run each candidate under identical cutoffs, horizons, and data versions.
Replay the operating decision for the strongest candidates.
Record results by segment, horizon, and period type, not only as averages.

Evaluate several historical forecast cycles

Use rolling-origin backtesting. Each cutoff should rebuild features from information available at that time, generate the complete required horizon, and compare the forecast with the later observed outcome. Include normal periods and difficult periods. Cover holidays, promotions, supply constraints, launches, closures, and unusual demand where the records are reliable. Store results by cutoff, horizon, product, location, and operating segment. One average score can hide a method that fails during the periods with the highest business cost.

Measure error, bias, and business cost

Use a small group of complementary measures. Translate the forecast into the operating decision when possible. Replaying stockouts, waste, holding cost, expedite cost, service breaches, overtime, or unused capacity can distinguish methods with similar statistical accuracy.

Mean absolute error reports error in the target unit.
Root mean squared error gives more weight to large misses.
Weighted absolute percentage error supports portfolio reporting when the denominator is stable.
Mean absolute scaled error and related scaled measures compare series with different volumes.
Bias shows repeated overforecasting or underforecasting.
Quantile or pinball loss evaluates probabilistic forecasts at a chosen service level.

Check hierarchy and aggregation

Many businesses forecast across product, location, channel, and time hierarchies. Decide where totals must agree. Compare bottom-up, top-down, middle-out, and reconciliation approaches against the levels where teams plan and report. Review accuracy and bias at each important level. Strong total accuracy can hide poor local decisions.

Plan for new and sparse items

New products, new locations, and slow movers lack the history statistical methods need. Decide the fallback before launch: a global model that shares information across the portfolio, a similar-item reference, a category-level forecast allocated downward, or a planner-set initial value with a review date. Measure these segments separately. A portfolio score can look healthy while every new item misses its launch plan.

Include overrides in the evaluation

Planner overrides can add information that is absent from the data. Store the original forecast, adjusted forecast, reason, author, time, scope, and actual outcome. Measure override value by reason and segment. Repeated useful overrides can identify a missing feature. Repeated harmful overrides can identify a training, interface, or accountability problem. Overrides also need a boundary. Define which roles may adjust which levels, how large an adjustment needs a second review, and when an override expires. Without boundaries the override layer becomes a second, undocumented forecasting system.

Test the operating constraints

The selected method must run inside the available time, cost, reliability, and support envelope. Review these operating factors: A smaller method with reliable operations can create more value than a higher-scoring method that misses the planning deadline or cannot be supported. Ask who answers when the forecast is wrong at month end. If nobody owns that answer, the method does not have an operating home yet, whatever its backtest score.

Training and inference duration
Compute and storage cost
Data freshness and publication deadline
Failure and fallback behavior
Model and feature versioning
Monitoring and delayed outcome collection
Explainability required by planners or reviewers
Skills required to diagnose and maintain the system
Runbooks, access controls, and ownership transfer

Common failure modes

Leakage in the backtest. A feature used information that arrived after the cutoff, and the winning method fails on live data.
One evaluation window. The method wins a calm year and loses the next promotion season.
Aggregate-only reporting. The portfolio average improves while the highest-margin items get worse.
Untracked overrides. Planners fix the forecast by hand, and the evaluation credits the model.
No re-review. The selected method runs for years while demand behavior, data, and costs change around it.

Select with a recorded decision

Record the chosen method, baseline, backtest cutoffs, data version, metrics, business loss, segments, assumptions, and operating constraints. Include the reason for rejecting close alternatives. The record lets a future reviewer understand the choice without re-running the trial. It also protects the team. When a method later underperforms, the record shows whether the original assumptions broke or the evaluation missed something. Set a review cadence. Demand behavior, product range, source data, commercial policy, and operating costs change. Method selection remains an operating decision after the first release.

Transfer ownership to the operating team

A selected method becomes part of the forecasting service. The evaluation record, data contract, monitoring thresholds, fallback rule, and retraining trigger belong with the team that will run it. Walk the operating team through a representative failure exercise: a late source, a training error, a missed publication deadline. The team should be able to explain what the system did, what the planner saw, and how service was restored. Schedule the first post-release review before launch. Compare observed error and bias with the backtest expectation at the same horizon and segment. Material differences usually trace to data availability, leakage in the trial, or a changed demand pattern.

Evaluation checklist

Which decision uses the forecast?
What grain, horizon, cadence, and lead time does the decision require?
Which inputs are available at every forecast cutoff?
Which baseline represents current performance?
How does the backtest prevent future information from entering past cutoffs?
Which measures reflect error, bias, and business cost?
Which hierarchy levels must agree?
How are sparse, intermittent, and new items handled?
How are planner overrides recorded and evaluated?
What happens when data, training, or publication fails?
Who owns monitoring, review, and method changes?

Related guide

The main demand forecasting guide compares method families, rolling backtests, hierarchy, monitoring after release, and the path from forecast to replenishment. For the complete method guide, read Demand forecasting techniques: how to choose and evaluate a method.