1233

CRITICAL APPRAISAL OF MACHINE LEARNING PROGNOSTIC MODELS FOR ACUTE PANCREATITIS: A SYSTEMATIC REVIEW

Date
May 21, 2024

Background:
Lack of an accurate prognostic tool for acute pancreatitis (AP) remains a critical knowledge gap. Machine learning (ML) techniques have been used to develop high-performing prognostic models in AP. However, methodologic quality has received little attention. High-quality reporting and study methodology are critical to model validity, reproducibility, generalizability, and clinical implementation. In collaboration with content experts in ML methodology, we performed a systematic review critically appraising ML AP prognostic models.
Methods:
Using a validated search strategy, we identified non-regression ML AP studies from the databases MEDLINE, PubMed, and EMBASE published between January 2021 and June 2023. Eligible studies included all that developed or validated new/existing ML models in patients with AP. We used the Prediction Model Risk of Bias Assessment Tool, a well-established tool, to assess risk of bias (ROB) in 4 domains (participants, predictors, outcomes & statistical analysis). Quality of reporting was assessed by the standards of the Transparent Reporting of a Multivariable Prediction Model of Individual Prognosis or Diagnosis – Artificial Intelligence (TRIPOD-AI) statement that recommends 27 standards to be reported in a ML prognostic model paper.
Results:
We identified 4240 studies of which 27 met the eligibility criteria. AP severity (40.7%) or mortality (22.2%) were the most common outcomes predicted. Studies originated from China (19), U.S (4), Hungary (2), Turkey (1), and New Zealand (1). All studies developed a new ML model (i.e., none externally validated an existing ML model). The mean area-under-the-curve for all models was 0.9 (SD 0.08), but ROB was high in at least one domain in all studies (Figure 1). In the statistical analysis domain, 89% of studies were at high ROB. Importantly, steps were rarely taken to minimize over-optimistic model performance (63% of studies). Studies reported on only 55.6% of the 27 items contained in the TRIPOD-AI statement with notable deficiencies in sample size justification (74.1%), data quality assessment (40.7%), and model updating techniques (66.7%). There were frequent omissions in model implementation considerations, such as human-AI interaction (88.9%), handling of low-quality or incomplete data (88.9%), and integration of models into the care pathway (55.6%). Additionally, there was a lack of reporting on source data (63%), analytical codes (92.6%), and study protocols (81.5%).
Conclusion:
Despite an expansion of newly developed ML prognostic models in AP, there are important limitations in the study design, reporting, and open science practice, undermining the models' validity, reproducibility, and generalizability. Multifaceted interdisciplinary efforts (content experts in AP and ML methodology) are needed to improve the rigor of studies that develop ML prognostic models in AP.

Tracks

Related Products

Thumbnail for EXOCRINE PANCREATIC INSUFFICIENCY INCIDENCE AT 12 MONTHS AFTER ACUTE PANCREATITIS: A PROSPECTIVE MULTICENTER STUDY
EXOCRINE PANCREATIC INSUFFICIENCY INCIDENCE AT 12 MONTHS AFTER ACUTE PANCREATITIS: A PROSPECTIVE MULTICENTER STUDY
BACKGROUND: Exocrine Pancreatic insufficiency (EPI) occurs following acute pancreatitis (AP) at variably reported rates, secondary to combination of suspected impairment in pancreatic enzyme secretion and damage to the pancreatic acinar cells…
Thumbnail for INCIDENCE OF EXOCRINE PANCREATIC INSUFFICIENCY AT 3 MONTHS AFTER ACUTE PANCREATITIS: A MULTICENTER PROSPECTIVE STUDY
INCIDENCE OF EXOCRINE PANCREATIC INSUFFICIENCY AT 3 MONTHS AFTER ACUTE PANCREATITIS: A MULTICENTER PROSPECTIVE STUDY
Background: The incidence of hypertriglyceridemia-associated acute pancreatitis (HTG-AP) has been increasing recently. Plasmapheresis can effectively remove triglyceride(TG) from plasma, but its clinical value in HTG-AP is unclear due to the lack of solid evidence…
Thumbnail for GERMLINE MULTIGENE PANEL TESTING IN ACUTE AND CHRONIC PANCREATITIS
GERMLINE MULTIGENE PANEL TESTING IN ACUTE AND CHRONIC PANCREATITIS
Germline genetic testing is recommended for younger patients with idiopathic pancreatitis, but there is no consensus recommendation for those over age 35. We aimed to analyze the results of genetic testing for pancreatic genetic mutations using a large dataset including all ages…
Thumbnail for DIFFERENTIATING PANCREATIC CYSTS THROUGH FLUID GLUCOSE UTILIZING A VALIDATED LABORATORY GLUCOSE TEST IN A PROSPECTIVE COHORT STUDY
DIFFERENTIATING PANCREATIC CYSTS THROUGH FLUID GLUCOSE UTILIZING A VALIDATED LABORATORY GLUCOSE TEST IN A PROSPECTIVE COHORT STUDY
Background: Cyst fluid glucose (CFG) is increasingly being used in the differentiation of pancreatic cystic lesions (PCLs)…