O3194: Protecting Adolescent Patient Privacy with Natural Language Processing

Background: The 21st Century Cures Act mandates the immediate release of health information to patients. While there are benefits to information sharing, careful consideration is required to protect patient privacy in the case of adolescents. Providing confidential care for adolescent patients around topics such as substance use, sexual health, and mental health is mandated by state law and is an important part of providing high quality health care for this population. In this presentation, we share the development and implementation of a natural language processing algorithm to support ongoing operational efforts to promote information sharing with our adolescent patients while preserving their privacy.

In preparation for the sharing of adolescent clinical notes, we launched a clinic-by-clinic documentation optimization project that involves (1) provider education around adolescent confidentiality, and (2) redesign of note templates to decrease bloat and reduce the risk of unintentional privacy breaches. Prior to each intervention, a manual audit of recent progress notes is performed to identify high-risk documentation practices and tailor the provided feedback. In order to improve the efficiency and impact of these manual audits, we sought to develop a natural language processing algorithm capable of identifying notes with confidential content.

Methods: We sampled 1,200 outpatient progress notes from visits with adolescent patients (aged 12 - 17 years) occurring between 2016 and 2019. These notes were annotated by a group of physicians to identify confidential content. Using an 80/20 test/train split and a TF-IDF (term frequency-inverse document frequency) transformation of the text, this corpus was used to train a logistic regression model that classifies whether a progress note contains confidential content. The algorithm was prospectively validated on a set of 240 clinical notes sampled from 2022. It was subsequently deployed into clinical operations to flag high-risk notes and identify confidential disclosures in an audit of one specialty clinic’s documentation.

Results: The prevalence of notes containing confidential content was 21% (255/1200) and 22% (53/240) in the train/test and validation cohorts. The logistic regression model achieved an AUROC of 88% and F1 score of 67% in prospective validation (Table). Using a threshold of 0.5, the model was used to flag notes from one of our subspecialty clinics in July 2022. During this month, 264 notes were written by 15 different providers. Of these, 60 notes were flagged as high-risk (Figure). One provider was identified as an outlier. A manual review of these notes identified frequent use of an automated phrase populated with the tobacco-use history of the patient.

Conclusion: An NLP algorithm can identify confidential content in progress notes with high accuracy. Its human-in-the-loop deployment in a pilot intervention identified outlier documentation practices and demonstrated efficiency gains over an otherwise fully manual note review.