We identify three specific areas of focus to advance human-centered AI:
• Designers and systems must understand the context of use and sense changes over time: Successful AI Engineering depends on the team’s ability to identify and articulate the desired system outcome and understand human and contextual factors affecting the outcome. The system itself must be able to learn when shifts in context have occurred. What are the best ways to maintain clarity around operational intent and mechanisms for adapting and evolving systems based on dynamic contexts and user needs?
• Development of tools, processes, and practices to scope and facilitate human-machine teaming: Implementation of AI systems entails high levels of interdependence between human and machine. Adoption of AI systems requires the primary users to interact with and understand systems, gaining appropriate levels of trust. Every AI system needs to be designed to recognize boundaries and unfamiliar scenarios, and to provide transparency regarding its limitations.
• Methods, mechanisms, and mindsets to engage in critical oversight: AI systems learn through data and observations, rather than being explicitly programmed for a deterministic outcome. Critical and reflective oversight by organizations, teams, and individuals that create and use AI systems is needed to uphold ethical principles and proactively consider the risks of bias, misuse, abuse, and unintended consequences through design, development, and ongoing deployment.
For each area, we identify ongoing work as well and challenges and opportunities in developing and deploying AI systems with confidence.
Publisher StatementThis material is based upon work funded and supported by the Department of Defense under Contract No. FA8702-15-D-0002 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center.
The view, opinions, and/or findings contained in this material are those of the author(s) and should not be construed as an official Government position, policy, or decision, unless designated by other documentation.
References herein to any specific commercial product, process, or service by trade name, trade mark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by Carnegie Mellon University or its Software Engineering Institute.
NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITYMAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT
Copyright StatementCopyright 2021 Carnegie Mellon University.
This material has been approved for public release
and unlimited distribution. Please see Copyright notice for non-US Government use
Internal use:* Permission to reproduce this material and to prepare derivative works
from this material for internal use is granted, provided the copyright and “No Warranty”
statements are included with all reproductions and derivative works.
External use:* This material may be reproduced in its entirety, without modification,
and freely distributed in written or electronic form without requesting formal
permission. Permission is required for any other external and/or commercial use.
Requests for permission should be directed to the Software Engineering Institute at
* These restrictions do not apply to U.S. government entities.