Is Explainable Medical AI a Medical Reversal Waiting to Happen? Exploring the Impact of AI Explanations on Clinical Decision Quality
2023-05
Loading...
View/Download File
Persistent link to this item
Statistics
View StatisticsJournal Title
Journal ISSN
Volume Title
Title
Is Explainable Medical AI a Medical Reversal Waiting to Happen? Exploring the Impact of AI Explanations on Clinical Decision Quality
Authors
Published Date
2023-05
Publisher
Type
Thesis or Dissertation
Abstract
Medical AI systems generate personalized recommendations to improve patient care, but to have an impact, the system recommendation must be different from what the clinician would do on their own. This impact might be beneficial (in the case of a high-quality recommendation), and clinicians should follow the system; on the other hand, AI will be wrong occasionally and clinicians should override it. Resolving conflict between their own judgment and the system’s is crucial to optimal AI-augmented clinical decision-making. To help resolve such conflict, there have been recent calls to design AI systems that provide explanations for the reasoning behind their recommendations, but it is unclear how system explanations affect how clinicians incorporate AI recommendations into care decisions. This dissertation explores this issue against the history of medical reversals—technologies and treatments that were initially thought beneficial but ultimately were ineffective or even harmful to patients. Ideally, AI explanations are helpful; however, practicing evidence-based medicine requires that we validate this normative statement to ensure that explanations are not either ineffective or worse, actively harmful. We employ mixed methods, combining semi-structured interviews and three computer-based experiments to examine factors posited to support proper system use. To ground the findings, Study 1 interviews with clinicians highlighted that disclosing the confidence level and explanations for AI recommendations might create new conflicts between the system and clinician. To evaluate this within the clinical context, we conducted a trio of experiments where clinical experts were faced with drug dosing decision tasks. In Study 2, participants received AI recommendations for drug dosing and were shown (or not) the confidence level and an explanation. In Study 3, participants were shown explanations (or not) and received two patient cases each with either a high- or low-quality AI recommendation. Contrary to theoretical predictions, providing explanations did not uniformly increase the influence of AI or improve clinical decision quality. Instead, explanations increased the influence of low-quality AI recommendations and decreased the influence of high-quality recommendations. In Study 4, participants were shown one of four types of explanations along with either a high- or low-quality recommendation. Again, explanations did not optimally improve the influence of AI or improve decision-quality. Two relevant findings help explain why. First, the initial disagreement between the users’ a priori judgment and the AI recommendation, along with the quality of the recommendation, were much more influential than any of our explanation formats. And second, the qualitative and mediation analyses indicate that clinicians do carefully consider explanations, but the way that explanations impact overall perceptions of explanation detail and helpfulness of the explanation or decision conflict do not necessarily explain the observed effects. We see indications that the individual information cues in the explanations are compared against the users’ own knowledge and experience, and the way discrepancies between the users’ own opinion about these cues and the conclusions the system purports to draw from them can influence use of the recommendation. Further work is needed to understand exactly how the information from explanations is shaping use of AI advice.
Description
University of Minnesota Ph.D. dissertation. May 2023. Major: Business Administration. Advisors: Shawn Curley, Yuqing Ren. 1 computer file (PDF); xxii, 251 pages.
Related to
Replaces
License
Collections
Series/Report Number
Funding information
Isbn identifier
Doi identifier
Previously Published Citation
Suggested citation
Clement, Jeffrey. (2023). Is Explainable Medical AI a Medical Reversal Waiting to Happen? Exploring the Impact of AI Explanations on Clinical Decision Quality. Retrieved from the University Digital Conservancy, https://hdl.handle.net/11299/258718.
Content distributed via the University Digital Conservancy may be subject to additional license and use restrictions applied by the depositor. By using these files, users agree to the Terms of Use. Materials in the UDC may contain content that is disturbing and/or harmful. For more information, please see our statement on harmful content in digital repositories.