The most frequent indications in level 1 ERCPs were CBD stones (n=251; 27.6%), benign strictures (n=216; 23.7%) or malignant strictures (n=240; 26.4%). Malignant stricture was in the level 2 group, the most frequent indication for ERCP; this accounted for 41.7% of the total number (n=96). The major part of level 3 ERCPs was performed for pancreatic indications, such as endoscopic therapy in chronic pancreatitis (47.2% of all level 3 ERCPs).

Procedural success was calculated for the most common indications: stone extraction and benign or malignant strictures. In 148 patients with stones, both sphincterotomy and stone extraction were attempted after cannulation of CBD. In these cases, sphincterotomy was successfully performed in all cases as well and complete stone extraction was successful in 92.3%. Procedural success in cases in which ERCP was indicated for suspicion of CBD stones was therefore 92.3% (range 84.2–94.4%). In patients with strictures (either benign or malignant; n=482), CBD cannulation success rate was 98.4% and stent placement was successful in 96.8% of the cases, resulting in an overall 95.3% (range 82.4–100%) procedural success rate.

The CBD cannulation success rate in patients with a virgin papilla compared with patients who had undergone a previous sphincterotomy was significantly different (87.0 (range 79.2–95.5) vs 98.4% (range 90.5–100); χ²=36.66, p<0.01). Analysing CBD cannulation success rate in previous ERCP failure versus no failed procedure before (86.7 vs 96.2%) showed a significant difference as well (χ²=13.88, p<0.01).

The participating endoscopists were asked to give their opinion about the self-assessment programme through a short questionnaire. The common opinion was that this programme was valuable and useful to gain insight into performance. The form was found to be easy to fill out with clear questions. It took the participating endoscopists approximately 1 min to fill out the form. The endoscopists were also asked for reasons why some procedures were not assessed. The main reason for this was that they forgot to fill out RAF-E due to time pressure or busy programmes. Receiving feedback on performance was stated as important in order to achieve and maintain good adherence.

The demand for quality assessment in endoscopic procedures is increasing, but up until now, there is still no gold standard to assess the quality of ERCPs. Dutch guidelines state that an endoscopist is certified for ERCP when he has performed 100 procedures (Dutch Association of Gastroenterologists). Obviously, there is no scientific basis for this threshold number and the quality of these procedures remains unknown. ASGE guidelines state that a CBD cannulation success rate of ≥90% is an overall appropriate target, including experts, and that most endoscopists should be able to achieve a success rate of 85%.4 In general, gastroenterologists in smaller regional hospitals usually perform ERCPs with a level 1 difficulty degree. Technically more challenging procedures with difficulty level 2 or 3 are more often performed in tertiary referral centres. Questions regarding quality are being raised, such as the number of successful CBD cannulations and whether there was a difference in success between virgin papillas and patients with previous sphincterotomy. Numbers on the most frequently performed types of procedures are interesting as well. Another matter might be about a case with successful CBD cannulation, but failed stone extraction. On top of these questions, we wanted to encourage critically reflecting on one's performance. Our aim was to develop a self-assessment form that was easy to fill out and addressed the problems raised. A major drawback of too extensive evaluation forms is the inversely correlated drop in adherence to filling them out as we experienced in a pilot study. As reported by the participants, the most common reason not to fill out the form was lack of time during a busy programme. It was thus necessary to find a balance between the number of questions asked in the form and ease of completing it. Unfortunately, this implies an inevitable trade-off for a number of parameters that would have been interesting to monitor as well. In order to achieve compliance as high as possible and to develop a practical assessment tool for future, the most important outcome parameters were determined through expert opinion and this process resulted in the development of the RAF-E form used in this study. The final version of the form combines registration of procedural intention and outcome; we have used parts of previously validated assessment tools and metrics as described by Peter Cotton.17 18 Parameters that were scored as ‘partially successful’ were considered to be failures in the analysis in order to avoid any discussions on definitions of partial success. The results have shown that RAF-E is an easy-to-use device that provides insight into performance of individual endoscopists as well as larger groups. Unfortunately, the improvement plan was sparsely filled out by the experts, so it is difficult to make a statement on whether performance has improved afterwards. However, it might be worth considering linking RAF-E to the electronic endoscopy reporting system. This is a desirable step to take within the near future, which ensures the reliability of this method and provides easy accessible data for analysis. Since too much registration seems to be a burden for doctors in general, as well as for endoscopists, we believe that this linking of systems will result in even more reliable data.

Previous studies on ERCP quality focused mainly on complications, an accepted outcome parameter to assess quality.19–21 However, next to complication registration, we believe that the procedural quality is of equal importance as stated above. Some study groups have published their performance data on, that is, cannulation success with numbers similar to our centre (92% to 94%).6 7 22 Since performing ERCP only for diagnostic purposes, for example, in suspected CBD stones23 24 is considered obsolete and risky, less invasive methods such as magnetic resonance cholangiopancreatography (MRCP) or endoscopic ultrasound (EUS) are recommended for diagnostics. ERCP should thus be limited to therapeutic interventions. Taking this into account, therapeutic or procedural success is an important outcome parameter as well. ASGE states that clearance of CBD stones should be achieved in >85% of the cases. In our centre the overall success rate was 86.8%, which is adequate. Stent placement was successful in 97.8% of patients, which is in line with the ASGE recommendations as well (>90% success rate). These data show that our RAF-E provides insight into performance criteria such as those formulated by ASGE. We used a modified version of the Schutz's classification in this study. An ASGE working party has proposed a revised complexity score with four difficulty categories. This score was not yet available at the start of this study.25 The results of our study are in line with the findings of the ERCP Quality Network,26 which is an electronic system where endoscopists can enter their data anonymously. As a result of this anonymity, no data verification can be performed and data are thus subject to bias. Moreover, one cannot recollect whether the ERCPs entered in the database are the only procedures performed by the participating endoscopists (registration bias). On the other hand, since the ERCP Quality Network is anonymous, there is less incentive to leave out failed cases. Our study was single centre; the ERCP Quality Network enables endoscopists from different centres to enter their procedures in the database.

This is the first study to show the rationale for using a self-assessment programme in order to assess the quality of ERCPs and the prospective design is one of the strengths. In the Netherlands, no methods are available to gain insight into quality and performance of ERCP apart from retrospective database research. This results in the unique character of our data.

There are some limitations to this study. One possible source of bias is registration bias. The endoscopists themselves are responsible for filling out the assessment forms correctly. One might state that the forms are sensitive for falsification since they are not linked to an endoscopy report database. An endoscopist could just leave out the procedures that failed. Therefore, reports of all ERCPs performed in the time frame were extracted to quantify and evaluate non-assessed procedures. These non-assessed procedures were checked manually on procedural outcomes such as CBD cannulation success. There were no significant differences in outcome between assessed and non-assessed procedures, which makes it unlikely that procedures were left out on purpose. Another limitation of this study is that it was performed in a single tertiary referral centre. The performance numbers cannot be extrapolated to general endoscopists in smaller hospitals with a different workload and case mix. Moreover, there were quite some variations in numbers and case mix between the endoscopists in this study alone. This might have had some influence on the results, but this is a reflection of the workload and caseload in our endoscopy unit and probably many other endoscopy units across the world. However, a clear relation between numbers of ERCPs performed and outcome has not been established yet.27 The numbers in this study are too small for a clear point of view on this topic, but when looking closer at our analysis, there was no correlation between volume and performance of the participating endoscopists. The one with the lowest volume was not the worst performer and vice versa. Finally, the impact of trainees on procedural outcome or success was not established in this study. Since this study was performed in a teaching hospital, trainees were involved in most ERCPs. It would be interesting to analyse their contribution in a new prospective study.

This study shows that a simple self-assessment form is a successful device to provide insight into quality of ERCPs on an individual basis as well as group performance. However, we experienced that in order to achieve and maintain a good adherence, reporting data to the participants on an individual basis was eminent, but this required time and dedication as well as one of the endoscopists stimulating the others to fill out the RAF-E forms. The next step is to roll out this self-assessment programme nationwide in the Netherlands, which will provide insight into quality and performance regarding ERCPs across the country. Next to investigating quality of ERCPs performed by experienced endoscopists, including trainees, in this self-assessment programme will provide additional information on learning curves on top of quality assessment.

Although quality monitoring of ERCPs is extremely important, useful assessment tools are lacking.