Dependability Benchmarking: making choices in an n-dimensional problem space

Dependability benchmarks should provide cost-effective ways to evaluate the behavior of components and computer systems in the presence of faults, allowing the quantification of dependability attributes or the characterization of the systems into well defined dependability classes. Beyond existing evaluation techniques, a dependability benchmark should represent an agreement accepted by the computer industry or/and by the user community, and specify the measures, the methods, and techniques required to perform measurements. This paper discusses the different dimensions involved in the dependability benchmarking problem and presents basic components required to specify dependability benchmarks. Although several obstacles still persist and are currently the subject of research, the definition of all the dimensions of the problem and the agreement of the community on a basic set of components that constitute a possible framework for dependability benchmarking seem to us a first step in the proposal of actual dependability benchmarks.