Ensuring Competency of Novice Laparoscopic Surgeons—Exploring Standard Setting Methods and their Consequences

Objective Simulation-based assessment tools have been developed to allow for proficiency-based simulator training in laparoscopy. However, few studies have examined the consequences of different standard setting methods or examined what level of proficiency is considered adequate for trainees. The objectives of the present study were to explore the consequences of different standard setting methods and to examine the proficiency level that surgical trainees are expected to reach, before performing supervised surgery on patients. Design Study participants undertook the Training and Assessment of Basic Laparoscopic Techniques test. The tests were video-recorded and rated using a simple scoring system based on number of errors and time. Participants were then asked to assess how high a score a novice should reach before performing supervised surgery on a patient. We then compared 3 methods of standard setting: expert performance level, contrasting groups method, and a modified Angoff method. Setting The study was conducted at the Copenhagen Academy for Medical Education and Simulation. The academy provides surgical simulation training in laparoscopy for trainees at the hospitals in the Capital Region and the Zealand Region of Denmark. Participants Participants were recruited among surgical trainees in their first year of specialty training from surgery, gynecology, and urology departments. A total of 40 participants were included and completed the trial. Results The different standard setting methods resulted in different pass/fail levels. At the expert performance level, the pass/fail level was 474 points—the contrasting groups method resulted in 358 points and the modified Angoff method resulted in 311 points among experienced surgeons, and 386 points among trainees. The different proficiency levels resulted in a failure rate of 0% to 50% of experienced surgeons and a pass rate of 0% to 25% of novices. Novice laparoscopic surgeons set a higher pass/fail level than experienced surgeons did (p = 0.008). Conclusion Required proficiency levels varies depending on the standard setting method used, which highlights the importance of using an established standard setting method to set the pass/fail level.