Inter- and Intra-rater Reliability and Accuracy of Periapical Radiography in the Diagnosis of Furcation Involvement in Maxillary Molars

Abstract

Inter- and Intra-Rater Reliability and Accuracy of Periapical Radiography in the Diagnosis of Furcation Involvement in Maxillary Molars Introduction: Studies have demonstrated the usefulness of limited field of view CBCT in determining proximity to anatomic structures and bone thickness, evaluating the health of the maxillary sinus, presence of root perforations, and the presence of the second mesiobuccal canal in maxillary molar. Prior to this study, there has been a deficit in literature evaluating the accuracy of periapical radiography as a diagnostic tool for furcation involvement in endodontic cases. Aim: The present study aims to evaluate inter- and intra-observer reliability and accuracy of periapical radiography in identifying furcation involvement in maxillary molars with CBCT imaging as a reference standard. Materials and Methods: Images of 81 maxillary molars were retrospectively selected from the radiographic archives of the University of Minnesota Division of Endodontics and 51 maxillary molars from the radiographic archives of a private practice limited to endodontics. Maxillary first and second molars with complete root development were selected based on the availability of a limited field of view CBCT volume and two diagnostic periapical images made on patients over the age of 18 years old. A total of 132 cases with the required images were screened for quality. Eighty-one sets of periapical images were viewed independently by one Board certified radiologist, one Board certified endodontist, one Board certified periodontist, and two second year residents receiving advanced training in endodontics. The images were rated by the five observers twice in a randomized order with two weeks between each viewing period to avoid recognition bias. For each maxillary molar evaluated, the evaluators were asked to rate the confidence level of the presence of a furcation involvement using the following scale: 1) definitely absent 2) uncertain 3) definitely present. If the observer marked uncertain, they were asked to provide an additional rating to indicate whether furcation involvement was 2a) absent with uncertainty or 2b) present with uncertainty. Intra- and inter-rater agreement for binary (No/Yes) ratings was assessed using the Fleiss’ kappa statistic. Rater agreement for 3-level (No/Uncertain/Yes) and 4-level (No/Uncertain No/Uncertain Yes/Yes) ratings was assessed using the weighted Fleiss’ kappa statistic with linear weights. Binary ratings were compared to the CBCT reference standard to calculate the accuracy, sensitivity, specificity, and positive and negative predictive values. Receiver operating characteristic analysis was performed on “Web-based Calculator for ROC Curves” (Johns Hopkins University School of Medicine, Baltimore, MD) (61) and R-based easyROC platform (62). The sensitivity and false positive fraction with each possible cut-off point for diagnosis (certain no, uncertain no, uncertain yes, certain yes), and the area under the curve (AUC) using a 4-point rating scale was calculated for each observer and in both sessions using the ROC software. The false positive fractions (x), and true positive fractions (y) for each observer from the analysis using a 4-point rating with various cut-off points were plotted in Microsoft Excel (Microsoft Corporation, Redmond, WA) to construct ROC curves for each session. AUC values were compared to commonly accepted interpretations of the area under the ROC curve (Table 4) to assess the accuracy of periapical radiography as a diagnostic test of furcation involvement (63). A Bonferroni test was performed for multiple comparison of each AUC for the session with the highest inter-rater kappa values (Session 2). Results: The average kappa value for inter-rater reliability using a 2-point rating scale was 0.52 and the kappa for a 3- or 4-point scale was 0.41 and 0.40, respectively. The overlapping confidence intervals of the kappa values suggest that raters had similar inter- and intra-rater reliability regardless of Board-certification in different specialties and varying experience levels. The average accuracy for periapical images for all observers was 0.64 with a sensitivity of 0.57 and a specificity of 0.84. A specificity of 0.84 suggests that a negative rating of a periapical radiograph is fairly specific in correctly identifying cases that truly do not have a furcation lesion in the reference standard. ROC analysis demonstrated that the area under the curve for each observer’s rating in each session was significantly different from the chance diagonal, suggesting that correct observations were not due to chance. In this study, the null hypothesis H0: AUC = 0.5 was rejected for each observer curve in both sessions (p<0.01) which suggests that periapical radiography is a discriminatory test with regards to diagnosis of furcation involvement of endodontic origin in maxillary molars. Conclusion: Within the limitations of this study, the following conclusions can be made. 1. Inter-rater agreement analysis suggests a moderate level of inter-rater agreement (kappa = 0.51 (0.40, 0.62) to 0.53 (0.41, 0.64)) for a 2-point rating scale. 2. Intra-rater agreement was moderate to substantial with kappa values 0.58 (0.39, 0.76) to 0.73 (0.58, 0.89) for a 2-point rating scale. 3. Receiver operating characteristic analysis suggests a fair to moderate level of discriminating ability of periapical radiography as a diagnostic tool for identifying furcation involvement in maxillary molars. 4. As a diagnostic tool, periapical radiography is better at correctly identifying the absence of furcation involvement than identifying the presence of furcation involvement in maxillary molars.