Abstracts

Rationale: People with epilepsy want to be informed about SUDEP (Sudden Unexpected Death in Epilepsy), but physicians do not commonly provide SUDEP counseling to their patients.  One possible intervention is to electronically prompt physicians in electronic medical records (EMRs) to provide counseling, particularly for high risk patients. Natural language processing (NLP) tools can automatically identify risk factors in physician notes of EMRs (Berry et al., 2017; Poster at AES), and could be used to generate electronic prompts based on the presence of risk factors. However, most NLP studies use data from a single institution, with limited evaluation of generalizability. An understanding of generalizability is needed if tools are to be disseminated on a larger-scale across hospitals for clinical use. The current study will evaluate: 1) the generalizability of NLP tools for detecting SUDEP risk factors, and 2) underlying barriers to generalizability. Methods: Data included 10,392 physician notes for 3,000 patients from one medical center (home) and 1,000 from five additional centers (away), all in New York City. Through chart review, we identified three risk factors, including: 1) generalized tonic clonic seizures, 2) refractory epilepsy, and 3) epilepsy surgery candidacy. Regular expressions, a type of NLP, were manually created with home training data. Sensitivity, positive predictive value, and F-measure were evaluated using home and away test data. Generalizability was defined as an absolute decrease in performance by less than 0.10 for away test data compared to home. To demonstrate how small revisions can improve generalizability, we removed three “boilerplate” standard phrases from away physician notes and repeated away performance. To evaluate underlying barriers to generalizability, we reviewed and qualitatively categorized errors in home and away test data. Results: Regular expressions performed well in home data (F-measure range=0.86-0.90), though performance degraded in away data (F-measure range=0.53-0.81). On first pass, generalizability was only achieved for 4 of 9 performance measures (44%). However, after removing three “boilerplate” standard phrases, away performance improved (F-measure range=0.79-0.89) and generalizability was achieved for nearly all measures. The main barrier to generalizability was use of standard text phrases, causing 104 of 171 errors (61%) in away test data. Differences in clinical terminology, physician note structure, and note formatting were not significant causes of errors in away physician notes. Conclusions: Regular expressions developed at one site performed well at five other sites, particularly after minor revisions to account for “boilerplate” standard phrases. Regular expressions are a feasible and generalizable approach to identify individuals for SUDEP counseling. Funding: Work is supported by generous funding from (1) Centers for Disease Control and Prevention (U01DP006089) and (2) the Epilepsy Research Fund (Epilepsy Research Risk Detection and Prevention of Death Fund).