The opportunities and cautions of cumulative sum analysis in assessing learning curves in pediatric urology

In this issue of the Journal of Pediatric Urology, Stern et al. present on an experienced laparoscopic surgeon's experience in adopting the robotic-assisted platform for pediatric pyeloplasty [ ]. This manuscript can be read at multiple levels. First, the authors nicely demonstrate both overall and task-specific learning curves for surgeons adapting the robotic technique. While the robotic platform has already saturated many marketplaces, this is relevant work for those areas where the surgical robot still represents an emerging opportunity. A second, and perhaps more generalizable, lens through which to view this work is as a road-map to cumulative sum analysis (CUSUM). CUSUM is a version of statistical process (or quality) control that may a novel concept to many in pediatric urology. In essence, this data analytic technique, seeks to monitor iterative trends over time and compare these outputs to a chosen standard. Specifically, CUSUM evaluates the cumulative sum of differences between the expected standard, herein described as the mean operative time to that point [ ]. This manuscript provides an opportunity to highlight both the power and limitations of the CUSUM analytic technique. First, one notes that CUSUM is not immediately intuitive, and I would encourage the reader to take the time in reviewing the authors' elegant description of the methodology in order to fully appreciate the manuscript. Owing to the underlying mathematics, CUSUM is markedly useful to measure small changes over time. To this end, CUSUM is an attractive tool to monitor operative learning curves in general, as one expects iterative and often smaller progressive improvements from case to case. The authors retroactively were able to define “cut-points” in proficiency, which could theoretically be used in association with this methodology to define surgical proficiencies in training progression and/or credentialing. Visually, one can see in Figure 1 how the learning curve decreases steadily, though not sequentially, after the first case (blue line). However, the CUSUM curve (black dots) demonstrates a much more visually dramatic series of breakpoints. One point of caution with the mathematics and reference value (operative time mean) is the tendency to convert outcomes that are progressing in a linear fashion into a more distinctly quadratic term, especially if the early experience tends to be greater than the overall sample mean, as is typical in operative times early in the learning curve [ ]. Put another way, any individual surgeon is likely to demonstrate a similar progression on a CUSUM plot, from learning to proficiency to mastery, based on use of their own means. A true comparison of mastery, however, would be benchmarking against a well-accepted standard. This comment is not to critique the surgeon-experience in the article, as they have progressed quite rapidly to commendable times, but rather to caution against over-interpretation of CUSUM in assessing learning curves more broadly. Alternatively, creating curves that benchmark against published complications, as has been reported in hypospadias [ ] and replicated in this manuscript, may provide a more broadly comparable use for CUSUM plots.