Analysis And Control Of Temporal Biases In Surgical Skill Evaluation

Abstract

Objectively and accurately assessing the technical skill of a surgeon is critically important. The current gold standard relies on a panel of expert surgeons evaluating surgical video footage using structured survey instruments. This is a prohibitively time-consuming process, thus leaving the majority of procedures unevaluated. Previous methods of evaluating skill remain prone to bias towards a surgeons' speed or task times, fueling the need to investigate the mechanisms underlying human motion in favor of techniques impervious to biases. The research objective of this work is to investigate the effects of time and speed on the relative accuracy of both human and computational methods of measuring surgical technical skill. Human methods consist of both expert and non-expert raters (faculty surgeons and Amazon Mechanical Turk crowd workers respectively). Computational methods consist of both neurophysiologically-derived measures from other disciplines and recent model-free machine learning methods. This research objective is pursued by the following four specific aims: Specific Aim 1: Determine whether surgical motion segments are directly correlated to tangential velocity prediction models, and if the result is impervious to surgeon speed. The objective was to test the null hypothesis that there is no relationship between the minimum jerk trajectory velocity prediction model and increases in technical skills proficiency. Prior work in human reaching suggests that adherence to the minimum jerk model should increase as technical skill increases, for proficiency of reaching motions in stroke rehabilitation. This thesis investigates whether this phenomenon holds true in surgical technical skill during simulated dry lab tasks. Specific Aim 2: Implement a classification algorithm which uses recorded data to classify surgical skill in a manner which is impervious to task time. It was hypothesized that recent machine learning algorithms which exploit temporal duration, used on kinematic data from dry lab laparoscopic training tasks, would increase the performance of the current state-of-the-art computational methods of classifying surgical skill. HASH(0x3de9178) Specific Aim 3: Determine how human ratings of surgical tasks are affected by video playback speed and duration. It was hypothesized that the perceived skill of a surgeon followed a unimodal function, in which human raters would experience an increase in perceived surgical skill as the speed of a surgical task video approaches the function's maximum, then immediately decreasing once being aware of the video's playback manipulation. Specific Aim 4: Measure the effect that pre-operative warm-up using validated virtual-reality simulator tasks has on practicing surgeons in real robotic surgeries as measured by the most accurate, least-biased methods detailed in the previous specific aims and prior art. This tested the hypothesis that pre-operative warm-up results in a measurable improvement in surgical technical skill among practicing surgeons (no novices) using surgical robots from live patients. This research concluded firstly that neurophsyiologically-derived models of skill, specifically the minimum jerk model, do not necessarily extend to surgical settings. Surprisingly this research found the opposite, that surgeon experts exhibit movements which deviate further from the minimum jerk model. Second, a classification algorithm was created, using a bidirectional long short-term memory network which controls for task time, and is capable of classifying experts and novices with over 95% accuracy for tasks most resembling real surgery. This research brought about questions of label noise and accuracy, and emphasizes the importance of properly labeled data for machine learning algorithms. It was found that humans appear to have a speed bias in rating surgeons both for laparoscopic surgical training tasks as well as real robotic surgery procedures. Unexpectedly, this effect was more substantial for more expert performances and negligible for novice performers. Counter to our original hypothesis, expectations derived from biological motion models -- that skill discrimination capability would unimodally decrease when video playback was obviously artificially sped up -- were not met. Observer ability to discriminate skill continues well after people are cognizant of a video being played at quicker speeds and no discernible difference between biological-motion-relevant question groups (e.g. motion fluidity) and other questions appeared. Finally, a new dataset of robotic surgeries was introduced, with 343 videos of robotic surgeries including tooltip kinematic data. Evidence obtained from motion metrics, crowd ratings, and faculty surgeon ratings suggest that no measurable warm-up effect was present in our population of 41 practicing surgeons; no evidence supported the use of warm-up.