From algorithms that steer autonomous vehicles and offer medical diagnoses to chatbots that answer tax queries, artificial intelligence (AI) offers the support that many facets of daily life require.
“Humans and machine algorithms have complementary strengths and weaknesses,” says study co-author Mark Steyvers (right), UCI professor of cognitive sciences, here with fellow co-author Padhraic Smyth, UCI Chancellor’s Professor of computer science. “Each uses different sources of information and strategies to make predictions and decisions.” Image Credit: Steve Zylius / UCI.
Developing smarter, more accurate tools necessitates a hybrid human-machine method, according to scientists at the University of California, Irvine. In a study published recently in the journal Proceedings of the National Academy of Sciences, they demonstrate a new mathematical model that can enhance performance by merging human and algorithmic predictions and confidence scores.
Humans and machine algorithms have complementary strengths and weaknesses. Each uses different sources of information and strategies to make predictions and decisions. We show through empirical demonstrations as well as theoretical analyses that humans can improve the predictions of AI even when human accuracy is somewhat below [that of] the AI—and vice versa. And this accuracy is higher than combining predictions from two individuals or two AI algorithms.
Mark Steyvers, Study Co-Author and Professor of Cognitive Sciences, University of California, Irvine
To test the method, scientists carried out an image classification experiment wherein human participants and computer algorithms worked individually to correctly identify inaccurate pictures of animals and everyday items — chairs, bicycles, bottles, trucks, etc.
The human participants graded their confidence in the accuracy of identifying each image as low, medium or high, while the machine classifier produced a continuous score. The results showed huge differences in confidence between AI algorithms and humans across images.
In some cases, human participants were quite confident that a particular picture contained a chair, for example, while the AI algorithm was confused about the image. Similarly, for other images, the AI algorithm was able to confidently provide a label for the object shown, while human participants were unsure if the distorted picture contained any recognizable object.
Padhraic Smyth, Study Co-Author and Chancellor’s Professor of computer science, University of California, Irvine
When predictions and confidence scores from both were pooled using the scientists’ new Bayesian framework, the hybrid model pointed to better performance than either human or machine predictions accomplished individually.
While past research has demonstrated the benefits of combining machine predictions or combining human predictions—the so-called ‘wisdom of the crowds’—this work forges a new direction in demonstrating the potential of combining human and machine predictions, pointing to new and improved approaches to human-AI collaboration.
Padhraic Smyth, Study Co-Author and Chancellor’s Professor of computer science, University of California, Irvine
This interdisciplinary project was enabled by the Irvine Initiative in AI, Law and Society. The union of cognitive sciences, which is concentrated on comprehending how humans think and act — with computer science — wherein technologies are created — will offer further insight into how humans and machines can work together to develop more accurate artificially intelligent systems, the scientists said.
Additional co-authors include Heliodoro Tejada, a UCI graduate student in cognitive sciences, and Gavin Kerrigan, a UCI Ph.D. student in computer science.
This study received funding from the National Science Foundation under award numbers 1927245 and 1900644 and the HPI Research Center in Machine Learning and Data Science at UCI.