ASSESSMENT OF A NOVEL ARTIFICIAL INTELLIGENCE-ENHANCED APPLICATION DESIGNED TO ASSIST THE CYTOLOGIC EVALUATION OF BILIARY STRICTURES: RESULTS OF A DOUBLE-BLINDED, PRAGMATIC TRIAL

Background: Several studies have demonstrated targeted next-generation sequencing (NGS) of biliary brushings and/or biopsy specimens can improve the identification of neoplastic bile duct strictures. However, these reports have largely been limited to retrospective analyses, single institutional experiences, and/or utilized NGS panels that were insufficiently comprehensive to account for the diversity of neoplasms that can cause a neoplastic stricture. The aim of this study was to prospectively evaluate NGS on a multi-institutional cohort of patients with a bile duct stricture in real-time.

Methods: Among seven institutions within a 3-year period, 1208 biliary specimens from 754 patients were submitted for targeted NGS that included two panels: a 28-gene NGS panel with 167 fusion genes (n=218) and a 161-gene NGS panel with 823 fusion genes (n=536). The NGS results were correlated with clinical presentation, history of primary sclerosing cholangitis (PSC), pathologic findings and follow-up. A stricture was designated as benign or neoplastic based on diagnostic pathology and/or a clinical course of >12 months.

Results: The sensitivity and specificity of NGS of biliary specimens for neoplasia were 82% and 96%, respectively. In comparison, pathologic evaluation had a sensitivity of 49% and a specificity of 100%. The combination of NGS and pathologic evaluation improved the sensitivity of both assays to 88%. Moreover, the addition of NGS to pathologic evaluation improved the sensitivity of biliary brushings from 36% to 84% and biliary biopsies from 47% to 87%. An expanded NGS panel (161 genes and 823 fusion genes) also showed improved sensitivity from 73% to 87% over a more limited panel (28 genes and 167 fusion genes). Repeat pathologic and NGS testing was performed for 118 patients and overall sensitivity increased to 91% but maintained a high specificity of 94%. Among 96 PSC patients, NGS had an 84% sensitivity as compared to pathologic evaluation with a 30% sensitivity. Of note, no statistically significant differences were observed in the performance of NGS based on the location of the patient’s stricture.

Conclusions: Through a prospective, multi-institutional study, the combination of NGS and pathologic evaluation of both biliary brushings and biopsy specimens improved the detection of neoplastic bile duct strictures. Furthermore, targeted NGS increased the sensitivity of identifying neoplasia in PSC patients.

Introduction: Cytologic evaluation of bile duct brushings (BDBs) is one of the most challenging tasks for cytopathologists, associated with low sensitivity and poor interobserver agreement. To address these difficulties, our group developed an artificial intelligence (AI) model to assist in the reading of BDB whole slide images (WSIs). This recent work demonstrated that AI-computer aided diagnosis (CADx) and AI-computer aided detection (CADe) tools matched the diagnostic accuracy of cytopathologists. The aim of this trial is to assess the efficiency and accuracy of an application housing the AI.

Methods: All consecutive BDB WSIs placed into a digital WSI database in October 2022 were identified, extracted, and uploaded to the application. The application scores each WSI as “positive” or “negative” for malignancy and then further prioritizes and ranks each WSI by the AI-CADx based on likelihood of malignancy. The AI-CADe prioritizes individual tiles/subregions by the likelihood that they contain malignant cytologic material. Users select any WSI and then evaluate the AI-prioritized list of tiles aided by the ability to freely navigate the WSI. Users label tiles as well as the entire WSI as “benign,” “atypical,” “suspicious,” or “malignant” (Figure 1A-B). User WSI labels of “benign,” “atypical,” and “suspicious” were considered “negative” and labels of “malignant” were considered “positive.” Users were blinded to all patient information.

AI-CADx, AI-enhanced cytopathologist interpretation (AI-CADe), and the official clinical cytopathologist interpretation (OCI) were compared. All WSI and patient data were reviewed by a multidisciplinary panel using a priori gold standard (GS) criteria to determine the presence or absence of malignancy for each WSI. The multidisciplinary panel was blinded to the user and AI-CADx interpretations.

Results: Of the 84 WSI uploaded, the panel identified 57 (67.9%) WSI as having a GS diagnosis labelled as “positive” (N=15, 26.3%) and “negative” (N=42, 73.7%). The GS WSIs were evaluated using the application by 4 blinded cytopathologists. While WSIs generated on average 141,950 tiles each, on average users evaluated only 10.6 tiles/WSI (10.2 tiles/neg WSI vs 11.7 tiles/pos WSI; p = 0.135), requiring only an average of 84.1 seconds of total WSI evaluation (79.3 seconds/neg WSI vs 93.7 seconds/pos WSI); p = 0.346). Overall accuracy of GS WSI interpretation for the OCI, AI-CADx, and AI-CADe were similar (80.7%, 75.4%, and 80.7%, respectively). Additional performance characteristics are reported in Table 1.

Discussion: Our trial shows that use of an AI-enhanced application allows cytopathologists to perform a targeted and abbreviated review of select WSI subregions, while maintaining their interpretive accuracy. In the future, user annotations of WSI and tiles could assist in training the AI and further improve performance.