Abstracts

Rationale: For EEG there is no "gold standard" interpretation, and thus the diagnostic accuracy of an interpreter can only be estimated. We describe a new method to estimate diagnostic accuracy that requires no reference standard and is based on efficiently extracting the sensitivity (s) and specificity (p) of the interpreter from multiple re-readings of a set of EEGs. It also takes into account possible correlations among misclassification errors. Methods: Each of N EEG recordings was interpreted twice (≥ 2 interpretations are allowed) and classified as normal (NL) or abnormal (AbNL). The interpretation pairs form a contingency table. To quantify the correlation of misclassification errors between interpretations we defined Cohen's kappas that represent the conditional covariance of the pairs of interpretations for all AbNL (κ_A) and NL (κ_N) studies. The prevalence of abnormality (Δ) was estimated directly from the proportion of AbNL interpretations. s and p were computed from the contingency table as a function of assumed values for κ_A and κ_N. The method was tested in a simulation and with a large dataset of real EEG interpretations. Results: The simulation used the following realistic parameters: Δ=60%, s=0.8, p=0.75; κ_A=κ_N= 0.1. Using these values, 3000 studies and two interpretations per study were randomly generated, and the accuracy estimated from the results. The simulation was repeated 500 times. The 95% CI for the estimated values of s and p were 1.3% and 6.3% of the true values (Figure). Modeling of misclassification correlations suggested that for most cases kappa values remain ≤ 0.1 and have a small effect on the estimated accuracy. In the Figure each point represents an estimated accuracy, and the true accuracy is at the intersection of the horizontal and vertical gray lines. The distribution of estimated values is indicated along each axis. The black curve is the distribution when the true values of kappa are used in the estimation. Dashed and dotted curves show the distributions corresponding to estimated accuracies when κ_A=κ_N= 0 and and κ_A=κ_N= 0.2 are used, respectively. The method was also applied to a pre-existing set of 227 EEGs recorded from ED patients with AMS. The official clinical interpretation was labeled EEG⁰. The EEGs were de-identified, technologist annotations removed, and re-interpreted by two of six randomly selected epileptologists (labeled EEG^1A,B). For this study all interpretations were recoded as Nl or AbNL. By using EEG⁰ as the reference standard (Δ=79%), s and p of EEG¹ were directly measured as s=0.87 and p=0.48. We then used our method to recalculate accuracy parameters from the contingency table of EEG¹ interpretations by assuming various values for κ_A and κ_N. The accuracy of EEG¹ was s=0.94 and p=0.77 (kappa= 0), and s=0.92 and p=0.67 (kappa= 0.2). Conclusions: These results from both simulated and actual EEG interpretations indicate that provided a realistic estimate of Δ, the method is able to estimate the true interpretation accuracy without recourse to a "gold" standard, and is capable of taking into account the correlation between misclassification errors.