UTILIZING CHATGPT FOR THE DEVELOPMENT OF A NATURAL LANGUAGE PROCESSING (NLP) ALGORITHM FOR THE AUTOMATED CALCULATION OF COLONOSCOPY QUALITY METRICS

Background: Accurately tracking quality metrics in endoscopy, such as the adenoma detection rate (ADR), is vital for GI practices. However, manually collecting data from free-text pathology reports is labor-intensive and impractical, especially within larger healthcare settings. Natural Language Processing (NLP), an AI technique, simplifies this task by streamlining data extraction from unstructured text. However, developing an NLP algorithm typically poses a challenge for physicians as it often requires substantial coding knowledge and usually requires collaboration with IT/computer programming specialists. Fortunately, AI language models like ChatGPT can guide non-programmers in algorithm development and code generation. The aim of this project was to harness ChatGPT’s capabilities to develop an NLP algorithm for calculating ADR and sessile serrated lesion detection rate (SDR) for screening colonoscopies performed at our large academic GI practice, serving both as an internal quality improvement effort and a scholarly demonstration of ChatGPT's practical application in medicine.

Methods: Using the data extraction feature within the ProVation MD endoscopy reporting software, a list of medical record numbers (MRNs) for patients who underwent screening colonoscopies across a three-year span was generated. Our institution's Epic Research & Data team extracted pertinent pathology reports associated with these MRNs, consolidating the data into a single spreadsheet. ChatGPT facilitated every step in developing the NLP algorithm, guided by specific prompts outlining goals and data structure. ChatGPT suggested using the open-source software Python for the task, and it provided detailed software setup instructions and wrote specific programming code (Figure 1) for ADR and SDR calculations, including overall and physician-specific metrics. ChatGPT assisted in troubleshooting code errors and formulating code revisions. Ultimately, the finalized NLP algorithm underwent testing against manual reviews by two trained human annotators of data from a sample of 530 patients.

Results: The developed NLP algorithm accurately calculated overall and physician-specific ADR and SDR with 99.1% precision compared to the manual review. After identifying five minor errors in the manual review and rectifying them, the algorithm matched the manual review with 100% accuracy and no discrepancies. The algorithm was then successfully run on the full 3-year dataset of 3707 patients (Figure 2).

Conclusion: AI large language models, such as ChatGPT, can significantly aid in constructing highly accurate NLP algorithms for automated calculation of important endoscopy quality metrics. These models empower physicians without IT/programming expertise to independently develop such algorithms, reducing reliance on professional software engineers for assistance.