Using Receiver Operating Characteristic Analysis to Evaluate Large-Change Forecast Accuracy

This paper applies receiver operating characteristics (ROC) analysis to M3 Competition, micro monthly time series for one-month-ahead forecasts. Using the partial area under the curve (PAUC) criterion as a forecast accuracy measure and paired-comparison testing via bootstrapping, we find that complex methods (AutomatANN, Flores-Pearce2, Forecast ProSmart FCS, and Theta) perform best for forecasting large declines in these time series, which tended as a group to decline over time. A regression model of PAUC on a judgmental index for forecast method complexity provides further confirming evidence. We also found that a combination forecast, consisting of the median value of the top three methods, to perform better than the component methods, although not statistically so. The classification of top methods matches that obtained using conventional forecast accuracy methods in the M3 Competition―complex methods forecast these series better than simple ones.