O3250: Ranking the Unrankable: A Systematic Approach to Incorporating Expert Intuition in Clinical Decision-making for Rare and Heterogeneous Diseases

Background: Artificial intelligence (AI) is increasingly utilized in medical decision-making and clinical practice guideline (CPG) development, but its effectiveness is limited in areas with little existing knowledge or research, such as pediatric musculoskeletal infection (MSKI). In such heterogeneous disease processes, expert experience-based intuition plays a significant role in clinical decision-making, yet is scarce in the literature. Collecting unbiased clinical intuition is essential for reliable AI decision-making.
Current methods of clinical consensus, like the Delphi process, have limitations including the need for large groups of experts and the influence of group dynamics. To address these limitations, we propose the development of a Clinical Intuitional Ordinal Hierarchy (CLIOH) using a rank order system with multiple individual responses. We will demonstrate the effectiveness of this approach by creating a rank order of MSKI outcomes, validated by the objective measurement of C-Reactive Protein (CRP). This rank order can be incorporated into a large language model to produce a CPG that accounts for the heterogeneous outcomes seen in MSKI.

Methods: CLIOH was developed using digital pair-wise surveys for 19 MSKI outcomes, with rank order established by analyzing the frequency of selection. Inter- and intra-observer variability were assessed using kappa statistics. The rank order was then applied to a dataset of 850 pediatric MSKI patients, with outcomes categorized based on established rank order and peak CRP values assessed. Statistical analysis was conducted to determine the association between rank order and CRP values, including ANOVA and T-tests.

Results: A panel of 7 pediatric orthopaedic surgeons and 3 pediatric infectious disease physicians participated in the survey to establish a rank order of 19 MSKI outcomes. The survey results showed strong agreement with inter- and intra-observer variability of 88.8% and 92.7%, respectively. When the rank order was applied to a retrospective cohort of 847 pediatric patients with MSKI, there was a positive correlation between peak CRP values (r2=0.795), shown in Figure 1. ANOVA analysis determined difference between mean CRP was significant (p < 0.001). Pairwise comparisons of means of appropriately powered categories (9 or more patients) are shown in Figure 2. There was also a lack of statistically significant differences between means of CRP of ranks 13-19 due to underpowering.

Conclusion: Our study has successfully established a reliable CLIOH for MSKI outcomes based on expert physician intuition. This provides a systematic approach for incorporating clinical intuition in situations where published data is lacking. The high level of agreement among the expert physicians and the positive correlation between the established rank order and CRP validate the utility of this approach. Next steps involve utilizing the rank order to develop AI systems with clinical intuition and generate CPGs that can incorporate such inputs, ultimately improving machine learning software for rare and heterogeneous disease processes.