Abstracts

Rationale:
For selected individuals with drug-resistant epilepsy, a well-established and effective treatment is surgery, with earlier surgical intervention associated with better outcomes. Machine learning and natural language processing algorithms can identify surgical candidates early in their epilepsy course; however, published techniques use “black box” machine learning models with little insight into what specific clinical factors are predictive. Here, we use natural language processing with interpretable machine learning models to investigate keywords present in clinical notes that can identify people with epilepsy who may benefit from epilepsy surgery.
Method:
We performed a retrospective study of patients within the RENYC (Rare Epilepsies in New York City) database, containing text of medical encounters for pediatric patients with seizures, epilepsy, and/or convulsions across 5 NYC medical centers from 2010-2014. Patients were identified in the following cohorts: epilepsy controlled with anti-epileptic drugs alone (control), treatment-resistant epilepsy with no surgery (intractable), and those who had epilepsy surgery (surgery). For the surgery cohort, we only included those with at least 1.5 years of clinical notes prior to surgery. Text was processed and converted into a data feature matrix (dfm) containing 1 to 3-gram tokens (Fig 1). The dfm was used to train two machine learning models, Naïve Bayes (nb) and Random Forests (rf), with 10-fold cross validation.
Results:
Our study included 251 patients in control, 1,092 in intractable, and 87 in epilepsy surgery groups. When comparing the text of the surgery cohort with the non-surgery cohorts using a Fisher’s exact test (N=174), a total of 23,113 tokens were statistically significant, p< 0.05. A token frequency dfm was used to train the nb and rf models yielding accuracies (mean ± standard deviation) of 74.3% ± 8.9 (nb) and 75.8% ± 8.5 (rf). When only including the top 100 predictors in the rf model, we obtained improved accuracies at 81.0% ± 5.9 for the top 100 frequency predictors and 87.0% ± 5.6 for binary predictors. A recursive feature elimination of the binary data resulted in a best subset of 25 predictors with an accuracy of 87.5% ± 7.7. The top 5 predictors were “drug_use”, “separate”, “list”, “instruction”, and “use”. A list of the 25 predictors graphed by variable importance are shown in Figure 2.
Conclusion:
To identify these clinical predictors, our initial analyses demonstrate that natural language processing techniques and interpretable machine learning models can be used to identify keywords in clinical notes that are associated with epilepsy surgery.  In ongoing work, we are further examining the machine learning models to identify key words that are both predictive and clinically meaningful.
Funding:
:This work was funded by the Centers for Disease Control and Prevention (CDC), grant U01DP006089.