In this report, we introduce an approach to performance evaluation that focuses on self-assessment. We find that this approach provides both greater information fidelity to satisfy performance assessment objectives and the enhanced realism that cyber operators desired in training and exercise (T&E) activities. We implement a popular incident response tool that enables team members to record their actions and thought processes and facilitate assessing the team’s abilities. To further validate our approach, we conducted a survey of participants who used the tool to gather qualitative feedback on its effectiveness. The results of this survey highlight the perceived improvements in realism, the usefulness of self-assessment tools, and the overall impact on team dynamics and individual growth. This combined approach provides valuable insights into team performance, enables best practices to be identified, supports the refinement of mitigation strategies, and fosters actionable feedback for learning. By promoting self-assessment within a realistic T&E environment, this method improves overall team performance in cybersecurity operations by maximizing feedback on individual skills and leadership competencies.
Publisher Statement
This material is based upon work funded and supported by the Department of Defense under Contract No. FA8702-15-D-0002 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. The view, opinions, and/or findings contained in this material are those of the author(s) and should not be construed as an official Government position, policy, or decision, unless designated by other documentation. References herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by Carnegie Mellon University or its Software Engineering Institute.
This report was prepared for the SEI Administrative Agent AFLCMC/AZS 5 Eglin Street Hanscom AFB, MA 01731-2100.
NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN "AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.
[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.Copyright Statement
Copyright 2024 Carnegie Mellon University.