Do risk calculators accurately predict surgical site occurrences?

Introduction: Current risk assessment models for surgical site occurrence (SSO) and surgical site infection (SSI) after open ventral hernia repair (VHR) have limited external validation. Our aim was to determine (1) whether existing models stratify patients into groups by risk and (2) which model best predicts the rate of SSO and SSI. Methods: Patients who underwent open VHR and were followed for at least 1 mo were included. Using two data sets - a retrospective multicenter database (Ventral Hernia Outcomes Collaborative) and a single-center prospective database (Prospective) - each patient was assigned a predicted risk with each of the following models: Ventral Hernia Risk Score (VHRS), Ventral Hernia Working Group (VHWG), Centers for Disease Control and Prevention Wound Class, and Hernia Wound Risk Assessment Tool (HW-RAT). Patients in the Prospective database were also assigned a predicted risk from the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP). Areas under the receiver operating characteristic curve (area under the curve [AUC]) were compared to assess the predictive accuracy of the models for SSO and SSI. Pearson's chi-square was used to determine which models were able to risk-stratify patients into groups with significantly differing rates of actual SSO and SSI. Results: The Ventral Hernia Outcomes Collaborative database (n = 795) had an overall SSO and SSI rate of 23% and 17%, respectively. The AUCs were low for SSO (0.56, 0.54, 0.52, and 0.60) and SSI (0.55, 0.53, 0.50, and 0.58). The VHRS (P = 0.01) and HW-RAT (P < 0.01) significantly stratified patients into tiers for SSO, whereas the VHWG (P < 0.05) and HW-RAT (P < 0.05) stratified for SSI. In the Prospective database (n = 88), 14% and 8% developed an SSO and SSI, respectively. The AUCs were low for SSO (0.63, 0.54, 0.50, 0.57, and 0.69) and modest for SSI (0.81, 0.64, 0.55, 0.62, and 0.73). The ACS-NSQIP (P < 0.01) stratified for SSO, whereas the VHRS (P < 0.01) and ACS-NSQIP (P < 0.05) stratified for SSI. In both databases, VHRS, VHWG, and Centers for Disease Control and Prevention overestimated risk of SSO and SSI, whereas HW-RAT and ACS-NSQIP underestimated risk for all groups. Conclusions: All five existing predictive models have limited ability to risk-stratify patients and accurately assess risk of SSO. However, both the VHRS and ACS-NSQIP demonstrate modest success in identifying patients at risk for SSI. Continued model refinement is needed to improve the two highest performing models (VHRS and ACS-NSQIP) along with investigation to determine whether modifications to perioperative management based on risk stratification can improve outcomes.