This report lists relevant questions that decision makers should ask of machine-learning practitioners before employing machine learning (ML) or artificial intelligence (AI) solutions in the area of cybersecurity. Like any tool, ML tools should be a good fit for the purpose they are intended to achieve. The questions in this report will improve decision makers’ ability to select an appropriate ML tool and make it a good fit to address their cybersecurity topic of interest. In addition, the report outlines the type of information that good answers to the questions should contain. This report covers the following questions:
- What is your topic of interest?
- What information will help you address the topic of interest?
- How do you anticipate that an ML tool will address the topic of interest?
- How will you protect the ML system against attacks in an adversarial, cybersecurity environment?
- How will you find and mitigate unintended outputs and effects?
- Can you evaluate the ML tool adequately, accounting for errors?
- What alternative tools have you considered? What are the advantages and disadvantages of each one?
Publisher StatementCopyright 2019 Carnegie Mellon University. All Rights Reserved.
This material is based upon work funded and supported by the Department of Defense under Contract No. FA8702-15-D-0002 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center.
The view, opinions, and/or findings contained in this material are those of the author(s) and should not be con-strued as an official Government position, policy, or decision, unless designated by other documentation.
References herein to any specific commercial product, process, or service by trade name, trade mark, manu-facturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by Carnegie Mellon University or its Software Engineering Institute.
This report was prepared for the SEI Administrative Agent AFLCMC/AZS 5 Eglin Street Hanscom AFB, MA 01731-2100
NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN "AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.