Please note: Linked content is NOT stored on Carnegie Mellon University and we can't guarantee its availability, quality, security or accept any liability.
Empirical Calibration of Time Series Monitoring Methods Using Receiver Operating Characteristic Curves
Any type of content formally published in an academic journal, usually following a peer-review process.
posted on 01.09.2008by Jacqueline Cohen, Samuel Garman, Wilpen Gorr
Time series monitoring methods, such as the Brown and Trigg methods, have the purpose of
detecting pattern breaks (or “signals”) in time series data reliably and in a timely fashion.
Traditionally, researchers have used the average run length statistic (ARL) on results from
generated signal occurrences in simulated time series data to calibrate and evaluate these
methods, with a focus on timeliness of signal detection. This paper investigates the receiver
operating characteristic (ROC) framework, well-known in the diagnostic decision making
literature, as an alternative to ARL analysis for time series monitoring methods. ROC analysis
traditionally uses real data to address the inherent tradeoff in signal detection between the true
and false positive rates when varying control limits. We illustrate ROC analysis using time series
data on crime at the patrol district level in two cities and use the concept of Pareto frontier ROC
curves and reverse functions for methods such as Brown’s and Trigg’s that have parameters
affecting signal-detection performance. We compare the Brown and Trigg methods to three
benchmark methods, including one commonly used in practice. The Brown and Trigg methods
collapse to the same simple method on the Pareto frontier and dominate the benchmark methods
under most conditions. The worst method is the one commonly used in practice.