A VIDEO-BASED PERORAL ENDOSCOPIC MYOTOMY ASSESSMENT TOOL (POEMAT) - DEVELOPMENT AND INITIAL VALIDATION

Background: Video analysis has emerged as a potentially strategy for performance assessment of specialized techniques where local expertise and mentorship are limited. The aim of this American Society of Gastrointestinal Endoscopy (ASGE) supported study was to develop a video based skill assessment tool specific for peroral endoscopic myotomy (POEMAT).
Methods: The POEM procedure was deconstructed into basic components on video analysis by an expert panel. Subsequently, a modified Delphi panel and 2 validation exercises were conducted with the goal of iteratively refining the tool. Twelve assessors used the final POEMAT version to grade 10 POEM videos performed by 2 skilled endoscopists. Fully crossed generalizability studies (G analysis) evaluated the contributions of assessors, POEM endoscopists performance and technical elements to reliability. A G coefficient of ≥0.7 was accepted as reliable for formative assessment in training or practice. A decision (D) study was then conducted to identify the number of assessors required to achieve a reliable mean assessment score ≥.70 on core technical elements (submucosal entry, submucosal tunneling, myotomy, mucosal closure, and hemostasis).
Results: Eight technical elements were initially generated from task deconstruction and included in the tool using the Delphi methodology. Subsequent iterative refinements including the addition and/or modification of descriptive anchors for each technical element, aimed to provide specific explanations of what constituted effective and ineffective performance. Ultimately, the final POEMAT version comprised of 9 technical elements each with descriptive anchors (Figure 1). Results from G analysis for POEMAT are shown Table 1. Differences in performance between endoscopists accounted for 0.8-24.9% of the observed variance across the categories. The G analysis attributed most variance to the interaction between endoscopist performance and the assessor (range from 63.2% to 90.1%), indicating lack of agreement between assessors across videos. D study results showed that ≤16 assessors would be needed for a reliability threshold of ≥.70 for 4 out of the 5 core technical elements.
Conclusions: We developed and validated the first instrument to assess skills of performing POEM. Development and refinement of this video-based tool followed a rigorous protocol. The initial validation showed a low variance between endoscopists, considered experts in this procedure. The next step is to apply the instrument among endoscopists with varying levels of expertise and practice settings to establish and further improve the discriminative validity of this tool.