Factors Affecting Appropriate Reliance on Artificial Intelligence Decision Support Systems

Many applications of AI require humans and AI advisors to make decisions  collaboratively; however, success depends on how appropriately humans rely on the AI agent.  We demonstrated an evaluation method for a platform that used neural network agents of varying  skill levels for the simple strategic game of Connect Four. We manipulated the presence,  sequence, skill, and information display of Artificial Intelligence (AI) advice in a strategy game  against another AI opponent that sometimes varied its skill to measure their effect on users’  performance.

Human agent teams outperformed unaided subjects with those receiving the AI  recommendations simultaneously achieving the best results. Although team performance was  higher and subjects improved during game play, there was little evidence of learning from their  AI advisors. AI reliability proved to be the greatest determiner of team performance with  subjects retaining trust in higher skilled advisors even in varied environments. Those with higher  numeracy demonstrated the highest ability to make use of AI advice including more detailed  output formats including ranking of choices and probabilities. More reliable AI agents correlated  to higher AI trust while higher self-confidence correlated to greater rejection of AI advice,  greater confidence in success, but slightly lower performance. 

The value of these human agent teams depended on AI reliability, users’ ability to extract  lessons from their advice, and users’ trust in that advice. Organizations implementing human  agent teams should conduct testing to know how well users appropriately rely on AI  recommendations.