Between Dec 19, 2024 and Jan 2, 2025, datasets can be submitted to DRUM but will not be processed until after the break. Staff will not be available to answer email during this period, and will not be able to provide DOIs until after Jan 2. If you are in need of a DOI during this period, consider Dryad or OpenICPSR. Submission responses to the UDC may also be delayed during this time.
 

Is Explainable Medical AI a Medical Reversal Waiting to Happen? Exploring the Impact of AI Explanations on Clinical Decision Quality

Loading...
Thumbnail Image

Persistent link to this item

Statistics
View Statistics

Journal Title

Journal ISSN

Volume Title

Title

Is Explainable Medical AI a Medical Reversal Waiting to Happen? Exploring the Impact of AI Explanations on Clinical Decision Quality

Published Date

2023-05

Publisher

Type

Thesis or Dissertation

Abstract

Medical AI systems generate personalized recommendations to improve patient care, but to have an impact, the system recommendation must be different from what the clinician would do on their own. This impact might be beneficial (in the case of a high-quality recommendation), and clinicians should follow the system; on the other hand, AI will be wrong occasionally and clinicians should override it. Resolving conflict between their own judgment and the system’s is crucial to optimal AI-augmented clinical decision-making. To help resolve such conflict, there have been recent calls to design AI systems that provide explanations for the reasoning behind their recommendations, but it is unclear how system explanations affect how clinicians incorporate AI recommendations into care decisions. This dissertation explores this issue against the history of medical reversals—technologies and treatments that were initially thought beneficial but ultimately were ineffective or even harmful to patients. Ideally, AI explanations are helpful; however, practicing evidence-based medicine requires that we validate this normative statement to ensure that explanations are not either ineffective or worse, actively harmful. We employ mixed methods, combining semi-structured interviews and three computer-based experiments to examine factors posited to support proper system use. To ground the findings, Study 1 interviews with clinicians highlighted that disclosing the confidence level and explanations for AI recommendations might create new conflicts between the system and clinician. To evaluate this within the clinical context, we conducted a trio of experiments where clinical experts were faced with drug dosing decision tasks. In Study 2, participants received AI recommendations for drug dosing and were shown (or not) the confidence level and an explanation. In Study 3, participants were shown explanations (or not) and received two patient cases each with either a high- or low-quality AI recommendation. Contrary to theoretical predictions, providing explanations did not uniformly increase the influence of AI or improve clinical decision quality. Instead, explanations increased the influence of low-quality AI recommendations and decreased the influence of high-quality recommendations. In Study 4, participants were shown one of four types of explanations along with either a high- or low-quality recommendation. Again, explanations did not optimally improve the influence of AI or improve decision-quality. Two relevant findings help explain why. First, the initial disagreement between the users’ a priori judgment and the AI recommendation, along with the quality of the recommendation, were much more influential than any of our explanation formats. And second, the qualitative and mediation analyses indicate that clinicians do carefully consider explanations, but the way that explanations impact overall perceptions of explanation detail and helpfulness of the explanation or decision conflict do not necessarily explain the observed effects. We see indications that the individual information cues in the explanations are compared against the users’ own knowledge and experience, and the way discrepancies between the users’ own opinion about these cues and the conclusions the system purports to draw from them can influence use of the recommendation. Further work is needed to understand exactly how the information from explanations is shaping use of AI advice.

Description

University of Minnesota Ph.D. dissertation. May 2023. Major: Business Administration. Advisors: Shawn Curley, Yuqing Ren. 1 computer file (PDF); xxii, 251 pages.

Related to

Replaces

License

Collections

Series/Report Number

Funding information

Isbn identifier

Doi identifier

Previously Published Citation

Other identifiers

Suggested citation

Clement, Jeffrey. (2023). Is Explainable Medical AI a Medical Reversal Waiting to Happen? Exploring the Impact of AI Explanations on Clinical Decision Quality. Retrieved from the University Digital Conservancy, https://hdl.handle.net/11299/258718.

Content distributed via the University Digital Conservancy may be subject to additional license and use restrictions applied by the depositor. By using these files, users agree to the Terms of Use. Materials in the UDC may contain content that is disturbing and/or harmful. For more information, please see our statement on harmful content in digital repositories.