Carnegie Mellon University

File(s) stored somewhere else

Please note: Linked content is NOT stored on Carnegie Mellon University and we can't guarantee its availability, quality, security or accept any liability.

Using Receiver Operating Characteristic Analysis to Evaluate Large-Change Forecast Accuracy

journal contribution
posted on 2010-03-25, 00:00 authored by Wilpen Gorr, Matthew Schneider
This paper applies receiver operating characteristics (ROC) analysis to M3 Competition, micro monthly time series for one-month-ahead forecasts. Using the partial area under the curve (PAUC) criterion as a forecast accuracy measure and paired-comparison testing via bootstrapping, we find that complex methods (AutomatANN, Flores-Pearce2, Forecast ProSmart FCS, and Theta) perform best for forecasting large declines in these time series, which tended as a group to decline over time. A regression model of PAUC on a judgmental index for forecast method complexity provides further confirming evidence. We also found that a combination forecast, consisting of the median value of the top three methods, to perform better than the component methods, although not statistically so. The classification of top methods matches that obtained using conventional forecast accuracy methods in the M3 Competition―complex methods forecast these series better than simple ones.




Usage metrics