A NOVEL SWITCHING-MULTIMODAL ARTIFICIAL INTELLIGENCE TO SIMULTANEOUSLY CONVERT DIFFERENT ENDOSCOPIC ENHANCEMENT MODALITIES FOR ACCURATE ASSESSMENT OF INFLAMMATION AND HEALING IN ULCERATIVE COLITIS

Background: Virtual Chromoendoscopy (VCE) has proven effective in predicting disease activity in Ulcerative Colitis (UC), although challenges persist regarding local availability and expertise. Artificial intelligence (AI) models applied to VCE have demonstrated a remarkable ability to rapidly, objectively, and accurately predict inflammation. Nonetheless, training machine algorithms across different enhancement modalities remains challenging. Hence, this study pioneers a novel machine model designed to simultaneously detect different VCE enhancement modalities and facilitate the transition between images to improve and standardise AI-based assessment of inflammation in UC.

Methods: Endoscopic videos from 302 UC patients recruited in the international real-life prospective PICaSSO study were analysed. The endoscopic assessment of the rectum and sigmoid colon was performed using WLE, iScan 2 and iScan 3 modalities (Pentax, Japan). In the study's first phase, a switching AI model that detects and converts images across different modalities was developed. A neural network (NN) to identify the acquisition modality of each frame was trained and tested with 1531 (510 WLE, 518 iScan 2, and 503 iScan 3) and 321 (103 WLE, 109 iScan 2, 109 iScan 3) randomly extracted frames, respectively. Subsequently, a CycleGAN model was trained with 900 images per modality to allow inter-modality image switching. In the second phase, 240 annotated videos (4605 frames) were selected, with endoscopic activity graded by experts using UCEIS for WLE and PICaSSO for VCE. Videos were switched to missing modalities and used to train a previously developed deep-learning model for inflammation assessment.¹ Four models were trained: three using a single modality as input and one combining all modalities. Model performance in predicting inflammation was assessed by computing accuracy, sensibility, specificity and AUC.

Results: The switching model showed a remarkable ability to classify and convert images across different endoscopic modalities, achieving a 92% NN classifier accuracy on the test set. The deep learning model showed a sensitivity of 80% (95%CI 59%-93%), specificity of 94% (95%CI 82%-99%), accuracy of 89% (95%CI 79%-95%) and AUC of 0.91 in predicting inflammation when combining images obtained through the switching model. This multimodal approach improved the performance of single-modality models.

Conclusion: This study introduces an innovative multimodal “AI-switching” model capable of accurately detecting and simultaneously switching between different endoscopic enhancement modalities. Combining the images obtained through this model enables precise assessment of inflammation in UC patients, exhibiting promising potential for application in clinical trials and clinical practice.

¹ Iacucci M et al. Endoscopy. 2023;55:332-341.