Inter- and Intraobserver Variation in the Assessment of Preoperative Colostograms in Male Anorectal Malformations: An ARM-Net Consortium Survey

Aim: Male patients with anorectal malformations (ARM) are classified according to presence and level of the recto-urinary fistula. This is traditionally established by a preoperative high-pressure distal colostogram that may be variably interpreted by different surgeons. The aim of this study was to evaluate the inter- and intraobserver variation in the assessment by pediatric surgeons of preoperative colostograms with respect to the level of the recto-urinary fistula.

Materials and Methods: Sixteen pediatric surgeons from 14 European centers belonging to the ARM-Net Consortium twice scored 130 images of distal colostograms taken in sagittal projection at a median age of 66 days of life (range: 4–1,106 days). Surgeons were asked to classify the fistula in bulbar, prostatic, bladder-neck, no fistula, and “unclear anatomy” example. Their assessments were compared with the intraoperative findings (kappa) for two scoring rounds with an interval of 6 months (intraobserver variation). Agreement among the surgeons' scores (interobserver variation) was also calculated using Krippendorff's alpha. A kappa over 0.75 is considered excellent, between 0.40 and 0.75 fair to good, and below 0.40 poor. Surgeons were asked to score the images in “poor” and “good” quality and to provide their years of experience in ARM treatment.

Results: Agreement between the image-based rating of surgeons and the intraoperative findings ranges from 0.06 to 0.45 (mean 0.31). Interobserver variation is higher (Krippendorff's alpha between 0.40 and 0.45). Years of experience in ARM treatment does not seem to influence the scoring. The mean intraobserver variation between the two rounds is 0.64. Overall, the quality of the images is considered poor. Images categorized as having a good quality result in a statistically significant higher kappa (mean: 0.36 and 0.37 in the first and second round, respectively) than in the group of bad-quality images (mean: 0.25 and 0.23, respectively).

Conclusions: There is poor agreement among experienced pediatric colorectal surgeons on preoperative colostograms. Techniques and analyses of images need to be improved in order to generate a homogeneous series of patients and make comparison of outcomes reliable.