EXCEED: enabling and excelling equitable evidence generation for digital health.

Abstract

The merger of technology and healthcare has given rise to a new era of evidence-based medicine. Digital health technologies (DHTs), defined in 2022 as “systems that use computing platforms, connectivity, software, and/or sensors for healthcare and related uses,” have become rapidly incorporated across medical specialties on a global scale. Randomized controlled trials (RCTs), considered the gold standard of evidence generation, are used by regulators to assess the safety and efficacy of proposed DHTs prior to translation into clinical practice. However, the ubiquity and diversity of DHTs present challenges in quantifying the current scope of DHT-enabled RCTs. To ensure a balance between innovation and safety in DHT-enabled RCTs, two critical elements must be comprehensively assessed: 1) the representativeness of participant populations in which DHTs are trialed, and 2) the impact of reporting heterogeneity on safe and equitable translation of findings. Recent advances in generative artificial intelligence present an opportunity for machine-assisted evidence synthesis leveraging large language models (LLMs). Prompt engineering uses natural language to instruct LLMs in performing tasks such as classification and information extraction. These capabilities can be applied to literature review tasks, such as article eligibility screening, full-text review, and adherence evaluations. However, preprocessing files and establishing standardized LLM techniques for real-world evidence generation inclusive of time, technology type, and health specialty has not previously been developed to analyze DHT-enabled RCTs. In this work, we propose the EXCEED framework to accelerate LLM-assisted evidence synthesis, achieving scientific rigor on par with human annotators. We then apply the framework to assess the rapid growth of DHT-enabled RCTs and assess the impacts of reporting heterogeneity on equitable evaluation of digital medicine. In the following chapters, we propose: 1) information extraction definitions for DHT-enabled clinical trials, 2) a feasibility study of multi-purpose LLMs evaluated on classification and information extraction tasks, 3) the EXCEED framework for LLM-assisted evidence synthesis application of the framework in two case studies, 4) a modified-Delphi study of expert recommendations for validation of DHT-enabled RCT reporting items, and 5) DHT-enabled RCTs reporting recommendations based on mixed-methods, data-drive evidence.