Introduction
Recent advancements in artificial intelligence (AI) have transformed the landscape of health care, through innovative technologies that improve clinical decision making and patient outcomes. Thus far, the potential of AI specifically in research has been understudied, especially using tools such as Chat Generative Pre-Trained Transformer (ChatGPT). This study explores the pivotal role of prompt engineering in optimizing data abstraction and analysis in gastroenterology research.

Methods
We utilized ChatGPT for various data analysis tasks. Various prompt engineering strategies were used to optimize the AI’s accuracy of results for the following tasks:
(1) Reviewing pathology reports for patients with Barrett’s esophagus to categorize pathology as cancer, high-grade dysplasia, or low-grade dysplasia, (2) Reviewing endoscopy reports for presence of high-quality exam metrics in assessment for BE such as use of narrow band imaging (NBI) or Seattle Protocol (SP) sampling. In the Data Analysis feature of ChatGPT Plus (v4.0), spreadsheets with de-identified data were uploaded into ChatGPT with chat prompts for data extraction and analysis. Data was analyzed and returned by ChatGPT and then manually verified for accuracy. When errors were found, prompts were used to re-train ChatGPT to re-analyze the data and attempt to correct errors.

Results
For task (1) in this study, a data set with 4,700 procedures was used. Prompt engineering was used for ChatGPT to learn pathology classifications in BE. (Figure 1) ChatGPT analyzed the data set in less than 30 seconds. Errors were pointed out to ChatGPT to re-code a new algorithm to correctly categorize patients. (Figure 2) Out of 4,700 procedures in the data set, ChatGPT identified 198 diagnoses of cancer (83% accuracy), 139 diagnoses of high-grade dysplasia (46% accuracy), and 118 diagnoses of low-grade dysplasia (87% accuracy). Notably, when ten random incorrectly classified pathology reports were directly entered into ChatGPT to be classified, the diagnoses provided by ChatGPT were 100% accurate. For task (2), out of 4,700 procedures in the data set, ChatGPT identified 788 cases with documentation of NBI with 100% accuracy but missed 72 (1.5%) procedures. For documentation of SP, ChatGPT identified 373 cases with 21 (0.4%) missed procedures.

Conclusion
Our findings highlight the potential of AI tools such as ChatGPT in enhancing the efficiency and automation of medical research. Although ChatGPT has robust potential to perform data analysis tasks, it is critical that researchers understand its limitations, including inaccuracies in data analysis (with large data sets) and the necessity for manual verification and prompt readjustment. Despite these, our study proves that prompt engineering is effective in navigating many research tasks which could save trainees and researchers hours of manual abstraction.

Figure 1. Prompt Engineering Example for Barrett’s Esophagus Data

Figure 2. Iterative ChatGPT Prompting to Correct Manual Errors in Large Data Sets

Kevin Shah

Appearances