P3A415: Optimizing a De novo Artificial Intelligence-based Medical Device Under a Predetermined Change Control Plan: Improved Ability to Detect or Rule out Autsim in General Pediatric Settings

Background: Autism can be reliably diagnosed at 18 months, yet significant diagnostic delays persist. User-friendly, data-driven autism diagnostics suitable for deployment in time-pressured primary care settings could support streamline autism diagnosis and help meet the rising demand for evaluations that has outpaced specialist capacity. Canvas Dx, the first FDA authorized diagnostic for autism, was developed and validated prospectively to empower healthcare professionals (HCPs) to act rapidly upon first developmental concerns. As an AI-based digital diagnostic, the Canvas Dx capabilities, if “unlocked,” include an ability to iteratively learn and improve with exposure to new data. The predetermined change control plan (PCCP) is an FDA proposed regulatory mechanism allowing AI-based devices to take advantage of such capabilities safely and compliantly. The Canvas Dx algorithm decision thresholds were optimized under a PCCP that was part of the de novo classification request granted by the FDA in 2021 (DEN200069). The procedure and results are presented in this analysis.

Methods: Decision threshold optimization was achieved using a repeated train/test validation procedure on a dataset of 722 children with concern for developmental delay, aged 18-72 months (28% autism, 22% neurotypical, 50% other developmental delay, mean age 3.6 years, 39% female); 504 (mean age = 3.6 +/- 1.2 years; mean 39.1% female) were used for threshold selection and 218 (mean age = 3.7 +/- 1.2 years; mean 39.0% female) for testing. The composition of the training and test sets remained representative of the intended use population. In 1000 repeats, 70% of samples were selected for threshold optimization and 30% for evaluation, ensuring that no training data appeared in the test set.

Results: Canvas Dx combines three distinct inputs (a caregiver questionnaire, analysis of two short home videos, and an HCP questionnaire) to produce either a positive for autism, negative for autism, or indeterminate for autism output, as well as a detailed report for each child that helps identify developmental strengths and challenges, and better inform next steps. With the optimized algorithm Canvas Dx had a PPV of 87.5% (95% CI [82.5%, 96.7%]) and NPV of 95.6% (95%CI [90.6%, 100%]) and a determinate rate (positive or negative for autism) of 66.5% (95% CI [62.5, 71.00]).

Conclusion: The optimized decision thresholds significantly increased Canvas Dx’s determinate rate to 66.5%, while maintaining predictive values comparable to those achieved by the device with the previous algorithm. With optimized decision thresholds, Canvas DX can support primary care providers to efficiently and accurately diagnose or rule out autism in a greater proportion of children with concern for developmental delay. Enhanced diagnostic capacity could allow a greater number of children to access autism services during the critical early years of high neuroplasticity where interventions have the greatest impact.