Validation of the ACS-NSQIP Risk Calculator: A Machine-Learning Risk Tool for Predicting Complications and Mortality Following Adult Spinal Deformity Corrective Surgery
========================================================================================================================================================================

* Katherine E. Pierce
* Bhaveen H. Kapadia
* Sara Naessig
* Waleed Ahmad
* Shaleen Vira
* Carl Paulino
* Michael Gerling
* Peter G. Passias

## Abstract

**Objective** To calculate the risk for postoperative complications and mortality after corrective surgery of adult spinal deformity (ASD) patients using the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) surgical risk calculator (SRC).

**Methods** Patients aged ≥18 years undergoing corrective surgery for ASD were identified. Current procedural terminology (CPT) codes of 22800, 22802, 22804, 22808, 22801, 22812, 22818, 22819, 22843, 22844, 22846, 22847, 22842, and 22845 were assessed if the patient had an *International Classification of Diseases Ninth Revision* (ICD-9) scoliosis diagnosis (737.00–737.9). Calculated perioperative complication risk averages via the ACS-NSQIP surgical calculator were compared with observed complication rates. Outcomes assessed were as follows: serious complication, any complication, pneumonia, cardiac complication, surgical site infection, urinary tract infection, venous thromboembolism, renal failure, readmission, return to operating room, death, discharge to nursing or rehabilitation, sepsis, and total length of hospital stay. Predictive performance of the calculator was analyzed by computation of the Brier score. A Brier score is the sum of squared differences between the binary outcome and the predicted risk and ranges from 0 to a maximum Brier score = (mean observed outcome)*(1-[mean observed outcome]). Values closer to 0 are suggestive of better predictive performance. Length of stay (LOS) was assessed with a Bland-Altman plot, which plots the average of observed LOS on the *x* axis and the difference between the observed and predicted LOS on the *y* axis.

**Results** A total of 9143 ASD patients (58.9 years, 56% females, 29.2 kg/m2) were identified; 36.9% of procedures involved decompression and 100% involved fusion. The means for individual patient characteristics entered into the online risk calculator interface were as follows: functional status (independent: 94.9%, partially dependent: 4.4%, totally dependent: 0.70%), 1.6% emergent cases, wound class (clean: 94.7%, clean/contaminated: 0.8%, contaminated: 0.5%, dirty/infected: 1.4%), American Society of Anesthesiologists class (I: 2.7%, II: 40.7%, III: 52.1%, IV: 4.6%, V: 0%), 5.1% steroid use for chronic condition, 0.04% ascites within 30 days prior to surgery, 1.73% systemic sepsis within 48 hours of surgery, 0.40% ventilator dependent, 3.2% disseminated cancer, 15.6% diabetes mellitus, 52.8% use of hypertensive medications, 0.3% congestive heart failure , 3% dyspnea, 21.4% history of smoking within 1 year, 4.3% chronic obstructive pulmonary disease, 0.7% dialysis, and 0.1% acute renal failure. Predictive of any 30-day postoperative complications ranged from 2.8 to 18.5% across CPT codes, where the actual rate in the cohort was 11.4%, and demonstrated good predictive performance via Brier score (0.000002, maximum: 0.101). The predicted and observed percentages for each of the 13 outcomes were assessed and their associated Brier scores and Brier maximums were calculated. Mean difference between observed and predicted LOS was 2.375 days (95% CI 9.895–5.145).

**Conclusions** The ACS-NSQIP SRC predicts surgical risk in patients undergoing ASD corrective surgery. This tool can be used as a resource in preoperative optimization by deformity surgeons.

**Level of Evidence** 3.

*   predictive analytics
*   spinal deformity

## INTRODUCTION

Postoperative complications following spine surgery have been demonstrated to be a potential source of excessive healthcare costs for hospital systems as well as patients. It has been shown to account for up to half of Medicare expenditures, with lumbar degenerative disc disorder being the most common indication for spine surgery.1–3 These costs directly impact spine surgeons, as the bundled payment plans begin to reshape reimbursements, leading to a concomitant increase in an emphasis on identifying any unanticipated costs associated with procedures, as well as more effective patient risk identification and stratification.4–6

These pressures have prompted the medical community to increase the development of predictive analytics in an attempt to mitigate postoperative patient morbidity and mortality and reduce overall healthcare costs.7–16 Discharge delays can markedly increase costs, attributable to inpatient charges for hospital bed, which in the United States has been estimated to be over $5000 per day.17 Furthermore, additional unplanned procedures due to complications and revision surgeries were found to increase mean costs by more than 87% compared to elective procedures.18 Preliminary studies have allowed surgeons to make an association between complications and specific patient-related factors, such as advanced age, diabetes, higher body mass index (BMI), American Society of Anesthesiologists (ASA) classification, and number of levels for surgery.12,17,19,20 However, the large variety in comorbidities and reported complications, from 14% to 71%, has made it a challenge to develop statistical hypotheses that are generalizable and allow for accurate predictive metrics.12

There is currently a paucity of studies evaluating such metrics in patients undergoing surgery for adult spinal deformity (ASD). The effective evaluation of potential risks and complications, morbidity, and mortality in the postsurgical period is an important component to ensure medical optimization of certain comorbidities and to further improve decision-making by spine surgeons. Therefore, the purpose of our study was to calculate the risk for postoperative complications as well as mortality following corrective surgery of ASD patients using the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) surgical risk calculator (SRC).

## MATERIALS AND METHODS

### Data Source

This study utilized the ACS-NSQIP database to identify factors associated with the development of hospital-acquired conditions in surgical spine patients associated with frailty status. Created by the US Department of Veterans Affairs, the dataset comprises more than 150 preoperative, intraoperative, and 30-day postoperative variables from patients in upwards of 250 urban and rural hospitals across the United States. Current procedural terminology (CPT) coding and *International Classification of Disease Ninth Revision* (ICD-9) codes can be used within the database to identify specific diagnoses and surgical procedures. The method of data collection by the ACS-NSQIP has been standardized by training surgical clinical reviewers who collect and audit the data from randomly assigned patients; this results in validated data displaying interrater reliability. Information about ACS-NSQIP is available at [https://www.facs.org/quality-programs/acs-nsqip/about](https://www.facs.org/quality-programs/acs-nsqip/about).

### Study Design

A retrospective analysis of the ACS-NSQIP database was performed from the years 2005 to 2016. Patients included were aged ≥18 years undergoing elective ASD surgeries, isolated with CPT and ICD-9 codes.13 CPT codes used in the ACS-NSQIP risk calculator included those pertaining to deformity of the spine: under arthrodesis procedures for spine deformity (22800–22819), under osteotomy procedures on the spine (22214), and under spinal instrumentation procedures on the spine (22842–22846). Patients with emergent cases, preoperative sepsis, pneumonia, open wounds, ASA class 4–5, prior surgery within 30 days, nonelective procedures, or wound class 2–4 were excluded. The online ACS-NSQIP risk assessment calculator was utilized for the study.

### Data Collection

Standardized data collection forms tracked demographic and surgical parameters. Complication data were also collected and closely monitored. Any complication was defined per the ACS-NSQIP as one of the following: superficial incisional surgical site infection (SSI), deep incisional SSI, organ space SSI, wound disruption, pneumonia, unplanned intubation, pulmonary embolism, ventilator <48 hours, progressive renal insufficiency, acute renal failure, urinary tract infection (UTI), stroke, cardiac arrest, myocardial infarction, deep venous thrombosis, or systemic sepsis. Additionally, the ACS-NSQIP defines a serious complication in this capacity as death, cardiac arrest, myocardial infarction, pneumonia, progressive renal insufficiency, acute renal failure, pulmonary embolism, deep venous thrombosis, deep incisional SSI, organ space SSI, systemic sepsis, unplanned intubation, and UTI.

### Risk Calculator Input and Outcomes

Each of the ASD-associated CPT codes available was entered into the NSQIP SRC interface ([https://riskcalculator.facs.org/RiskCalculator/PatientInfo.jsp](https://riskcalculator.facs.org/RiskCalculator/PatientInfo.jsp)). For the question, “Are there other potential treatment options?” we selected “none.” “Surgeon Adjustment of Risks” was not used as a part of the initial patient risk assessment for this study. We documented the predicted quality outcomes and estimated risk percentage for each perioperative adverse event, including serious complications, any complications, pneumonia, cardiac complications, SSI, UTI, venous thromboembolism, and renal failure, readmission, return to operating room (OR), death, discharge to nursing or rehabilitation, and sepsis, as well as length of stay (LOS). These predicted estimates from the calculator were compared with the actual outcomes available within the ASD cohort of NSQIP patients.

### Statistical Analysis

Overall performance of the model was assessed by computing Brier scores for each outcome. The Brier score is a quadratic scoring rule to measure the distance between observed and predicted risk. It is calculated as the sum of squared differences between the binary outcome (*Y*) and the predicted risk (*p*): (*Y* − *p*).2 It ranges from 0 to a maximum value calculated as per the incidence of the observed outcome (*p*). Values closer to 0 are suggestive of better predictive performance and values closer to maximum are suggestive of a noninformative model ( < 0.05 considered poor). The maximum Brier score is calculated as Brier max = mean (*p*) × (1 − mean(*p*)). For example, a 50% (*P* = 0.5) observed incidence would yield a maximum Brier score of 25 for a noninformative model.

### Bland-Altman Plot

LOS (continuous) was assessed using a Bland-Altman plot by plotting the average of predicted and observed LOS on the *x* axis and the difference between the 2 on the *y* axis.

## RESULTS

### Cohort Overview

Overall, 9143 operative ASD patients were isolated in the NSQIP database. The cohort had a mean age of 58.9 years and a mean BMI of 29.2 kg/m2; 56% were women. Also, 5.2% of the population had a history of smoking in the past year. The average Charlson Comorbidity Index was 1.44, 36.9% of procedures involved decompression, and 100% involved fusion.

### Observed Outcomes in the ASD Cohort

The means for individual patient characteristics entered into the online risk calculator interface are as follows: functional status (independent: 94.9%, partially dependent: 4.4%, totally dependent: 0.70%), 1.6% emergent cases, wound class (clean: 94.7%, clean/contaminated: 0.8%, contaminated: 0.5%, dirty/infected: 1.4%), ASA class (I: 2.7%, II: 40.7%, III: 52.1%, IV: 4.6%, V: 0%), 5.1% steroid use for chronic condition, 0.04% ascites within 30 days prior to surgery, 1.73% systemic sepsis within 48 hours of surgery, 0.40% ventilator dependent, 3.2% disseminated cancer, 15.6% diabetes mellitus, 52.8% use of hypertensive medications, 0.3% congestive heart failure, 3% dyspnea, 21.4% history of smoking within the past year, 4.3% chronic obstructive pulmonary disease, 0.7% dialysis, and 0.1% acute renal failure (Table 1).

View this table:
[Table 1](https://www.ijssurgery.com/content/15/6/1210/T1)

Table 1 
National Surgical Quality Improvement Program (NSQIP) calculator factors for patients in the adult spinal deformity (ASD) cohort.

Overall, the rate of 30-day complications in the ASD cohort was 11%, with 10.1% presenting with a serious complication. Table 2 summarizes the incidence of observed 30-day operations in our cohort of ASD patients, as well as the predicted rates of complications averaged from the CPT codes utilized.

View this table:
[Table 2](https://www.ijssurgery.com/content/15/6/1210/T2)

Table 2 
Predicted percentages from the NSQIP calculator and observed percentages from the actual cohort of perioperative outcomes, as well as the Brier score and Brier max between the predicted and observed percentages.

### Predictive Performance for 30-Day Outcomes

Predicted rates via the ACS-NSQIP calculator of any 30-day postoperative complications ranged from 2.8% to 18.5% across CPT codes, where the actual rate in the cohort was 11.4%, and demonstrated good predictive performance via Brier score (0.000002, max: 0.101), as all other outcomes were assessed. All Brier scores and Brier maxes can be viewed in Table 2.

### Length of Stay

Predictive performance for LOS was assessed using the Bland-Altman plot (Figure). Overall, mean difference between observed and predicted LOS was 2.375 days with a 95% CI of 9.895 to −5.145.

![Figure](https://www.ijssurgery.com/http://www.ijssurgery.com/content/ijss/15/6/1210/F1.medium.gif)

[Figure](https://www.ijssurgery.com/content/15/6/1210/F1)

Figure 
Bland-Altman plot demonstrating predictive performance for hospital length of stay.

## DISCUSSION

As the number of spinal arthrodesis procedures continues to increase, there is a concomitant increase in associated healthcare costs, with estimates of a net increase in costs from 2001 to 2010 by more than $35 billion.4,21 Thus, there is an ever increasing need for patient risk identification and stratification in an attempt to reduce postoperative complications and overall healthcare expenditures. The results of our study showed that the ACS-NSQIP risk calculator in patients undergoing ASD correction was effective at predicting postoperative surgical complications. The implications of these results lend evidence to the utility of this tool for preoperative patient optimization.

The use of various databases, such as ACS-NSQIP, has been effective in the development of predictive algorithms for the assessment of a wide range of metrics, from postoperative complications to discharge prediction, as well as opiate use.2,8,17,22,23 A study by Mosher et al evaluated 207 patients undergoing pathologic humerus fracture fixation and the predictive accuracy of the ACS-NSQIP SRC. When this tool was compared to actual complications reported, there was no statistically significant difference (*P* < 0.1) in predicting serious adverse events, mild adverse events, cardiac complications, acute renal failure, pneumonia, SSIs, discharge to acute care facility, readmissions, reoperations, and mortality.24 Similarly, a study by Broda et al developed a risk prediction model in elective spine surgery by using a retrospective cohort of 177 928 patients using the ACS-NSQIP database.25 This study identified 16 independent risk factors that were statistically significant at predicting outcomes, with complication rates increasing as a function of the model scores.

However, other studies using the ACS-NSQIP risk calculator have demonstrated less effective predictability. A study by McCarthy et al applied the calculator to a retrospective cohort of 237 lumbar and 404 cervical spine surgery patients and found no significant outcome prediction in the lumbar cohort.26 This inability to detect a difference may have been due to an insufficient sample size, as the cervical cohort was able to significantly (*P* < 0.001) predict any complication as well as discharge to a skilled nursing facility.26,27 The studies on spine surgery have been typically underpowered to demonstrate true efficacy of the ACS-NSQIP risk calculator at predicting postoperative outcomes. However, there are several independent risk calculators being utilized that have showed promise in predicting spine surgery outcomes and can be used in conjunction with the risk calculator, or to further validate its usefulness in spine surgery.

There were some limitations to our study, primarily with the use of a retrospective database. Furthermore, although the risk calculator has been shown to be effective at predicting postoperative outcomes in various surgical specialties, its use in spine surgery has not been fully validated. There are also several known limitations with the use of the NSQIP database, such as the dependence on ICD-9 and CPT codes for the identification of patients and outcome metrics.28 Additionally, 1 marked limitation of the database is that patients are only tracked for 30 days after the index procedure, which means there may be additional long-term complications that are not tracked and therefore missed. However, we feel that the use of this risk calculator on a larger database is the most effective way to validate the use for patients preoperatively.

As bundled payment plans begin to change the reimbursements in an attempt to reduce healthcare costs, there is an even more importance in preoperative patient risk prediction.29 Many postoperative complications can result in marked morbidity and mortality; and often result in prolonged hospitalizations, readmissions, and/or reoperations.30,31 Our study found that the ACS-NSQIP risk calculator was predictive of postoperative risks in patients undergoing ASD surgery. This tool can be used for preoperative patient optimization and help mitigate and reduce postoperative complications. Furthermore, as the opioid epidemic continues to increase in the United States, there is additional utility for the application of a risk predictor to determine pain treatments in opioid-naïve patients.22 As more studies are conducted, the ACS-NSQIP risk calculator could be expanded to include additional metrics and be developed to make it a more accurate and precise tool for various surgical subspecialties. Further prospective randomized clinical studies are needed to validate this tool as a potential preoperative risk predictor.

## CONCLUSION

The ACS-NSQIP SRC predicts surgical risk in patients undergoing ASD corrective surgery. This tool can be used as a resource in preoperative optimization by spinal deformity surgeons.

## Footnotes

*   Funding The author(s) received no financial support for the research, authorship, and/or publication of this article.

*   Declaration of Conflicting Interests The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

*   This manuscript is generously published free of charge by ISASS, the International Society for the Advancement of Spine Surgery. Copyright © 2021 ISASS. To see more or order reprints or permissions, see [http://ijssurgery.com](http://ijssurgery.com).

## References

1.  1. Cutler DM , Ghosh K . The potential for cost savings through bundled episode payments. N Engl J Med. 2012;366(12):1075–1077. doi:10.1056/NEJMp1113361 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1056/NEJMp1113361&link_type=DOI) 
    
    [PubMed](https://www.ijssurgery.com/lookup/external-ref?access_num=22435368&link_type=MED&atom=%2Fijss%2F15%2F6%2F1210.atom) 
    
    [Web of Science](https://www.ijssurgery.com/lookup/external-ref?access_num=000301769700006&link_type=ISI) 

2.  2. Karhade AV , Ogink P , Thio Q , et al . Development of machine learning algorithms for prediction of discharge disposition after elective inpatient surgery for lumbar degenerative disc disorders. Neurosurg Focus. 2018;45(5):E6. doi:10.3171/2018.8.FOCUS18340 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.3171/2018.4.FOCUS18147&link_type=DOI) 
    
    [PubMed](https://www.ijssurgery.com/lookup/external-ref?access_num=30064328&link_type=MED&atom=%2Fijss%2F15%2F6%2F1210.atom) 

3.  3. Ugiliweneza B , Kong M , Nosova K , et al . Spinal surgery: variations in health care costs and implications for episode-based bundled payments. Spine. 2014;39(15):1235–1242. doi:10.1097/BRS.0000000000000378 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1097/BRS.0000000000000378&link_type=DOI) 

4.  4. Harada GK , Basques BA , Samartzis D , Goldberg EJ , Colman MW , HS A . Development and validation of a novel scoring tool for predicting facility discharge after elective posterior lumbar fusion. Spine J. 2020;20(10):1629–1637. doi:10.1016/j.spinee.2020.02.014 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1016/j.spinee.2020.02.014&link_type=DOI) 

5.  5. Dietz N , Sharma M , Alhourani A , et al . Bundled payment models in spine surgery: current challenges and opportunities, a systematic review. World Neurosurg. 2019;123:177–183. doi:10.1016/j.wneu.2018.12.001 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1016/j.wneu.2018.12.001&link_type=DOI) 

6.  6. Schoenfeld AJ , Harris MB , Liu H , Birkmeyer JD . Variations in medicare payments for episodes of spine surgery. Spine J. 2014;14(12):2793–2798. doi:10.1016/j.spinee.2014.07.002 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1016/j.spinee.2014.07.002&link_type=DOI) 
    
    [PubMed](https://www.ijssurgery.com/lookup/external-ref?access_num=25017141&link_type=MED&atom=%2Fijss%2F15%2F6%2F1210.atom) 

7.  7. Ames CP , Smith JS , Pellisé F , et al . Development of predictive models for all individual questions of SRS-22R after adult spinal deformity surgery: a step toward individualized medicine. Eur Spine J. 2019;28(9):1998–2011. doi:10.1007/s00586-019-06079-x 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1007/s00586-019-06079-x&link_type=DOI) 

8.  8. Horn SR , Passias PG , Oh C , et al . Predicting the combined occurrence of poor clinical and radiographic outcomes following cervical deformity corrective surgery. J Neurosurg Spine. 2019;32(2):182–190. doi:10.3171/2019.7.SPINE18651 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.3171/2019.7.SPINE18651&link_type=DOI) 

9.  9. Jain D , Singh P , Kardile M , Berven SH . A validated preoperative score for predicting 30-day readmission after 1-2 level elective posterior lumbar fusion. Eur Spine J. 2019;28(7):1690–1696. doi:10.1007/s00586-019-05937-y 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1007/s00586-019-05937-y&link_type=DOI) 

10. 10. Karhade AV , Ogink PT , QCBS T , et al . Machine learning for prediction of sustained opioid prescription after anterior cervical discectomy and fusion. Spine J. 2019;19(6):976–983. doi:10.1016/j.spinee.2019.01.009 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1016/j.spinee.2019.01.009&link_type=DOI) 
    
    [PubMed](https://www.ijssurgery.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fijss%2F15%2F6%2F1210.atom) 

11. 11. McGirt MJ , Sivaganesan A , Asher AL , Devin CJ . Prediction model for outcome after low-back surgery: individualized likelihood of complication, hospital readmission, return to work, and 12-month improvement in functional disability. FOC. 2015;39(6):E13. doi:10.3171/2015.8.FOCUS15338 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.3171/2015.8.FOCUS15338&link_type=DOI) 

12. 12. Osorio JA , Scheer JK , Ames CP . Predictive modeling of complications. Curr Rev Musculoskelet Med. 2016;9(3):333–337. doi:10.1007/s12178-016-9354-7 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1007/s12178-016-9354-7&link_type=DOI) 

13. 13. Passias PG , Oh C , Horn SR , et al . Predicting the occurrence of complications following corrective cervical deformity surgery: analysis of a prospective multicenter database using predictive analytics. J Clin Neurosci. 2019;59:155–161. doi:10.1016/j.jocn.2018.10.111 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1016/j.jocn.2018.10.111&link_type=DOI) 

14. 14. Quddusi A , Eversdijk HAJ , Klukowska AM , et al . External validation of a prediction model for pain and functional outcome after elective lumbar spinal fusion. Eur Spine J. 2020;29(2):374–383. doi:10.1007/s00586-019-06189-6 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1007/s00586-019-06189-6&link_type=DOI) 

15. 15. Safaee MM , Scheer JK , Ailon T , et al . Predictive modeling of length of hospital stay following adult spinal deformity correction: analysis of 653 patients with an accuracy of 75% within 2 days. World Neurosurg. 2018;115:e422–e427. doi:10.1016/j.wneu.2018.04.064 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1016/j.wneu.2018.04.064&link_type=DOI) 
    
    [PubMed](https://www.ijssurgery.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fijss%2F15%2F6%2F1210.atom) 

16. 16. Staartjes VE , de Wispelaere MP , Vandertop WP , Schröder ML . Deep learning-based preoperative predictive analytics for patient-reported outcomes following lumbar discectomy: feasibility of center-specific modeling. Spine J. 2019;19(5):853–861. doi:10.1016/j.spinee.2018.11.009 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1016/j.spinee.2018.11.009&link_type=DOI) 

17. 17. Ogink PT , Karhade AV , QCBS Thio , et al . Development of a machine learning algorithm predicting discharge placement after surgery for spondylolisthesis. Eur Spine J. 2019;28(8):1775–1782. doi:10.1007/s00586-019-05936-z 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1007/s00586-019-05936-z&link_type=DOI) 

18. 18. Jacobs K , Dewilde T , Vandoren C , et al . Variability in hospital costs of adult spinal deformity care. Spine. 2020;45(17):1221–1228. doi:10.1097/BRS.0000000000003497 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1097/BRS.0000000000003497&link_type=DOI) 

19. 19. Cho CW , Kwon CHD , Kim JM , Choi GS , Joh JW , Lee SK . Comparative analysis of the clinical outcomes of liver transplantation for probable and definite auto-immune hepatitis by international diagnostic scoring criteria. Transplant Proc. 2017;49(5):1126–1128. doi:10.1016/j.transproceed.2017.03.014 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1016/j.transproceed.2017.03.014&link_type=DOI) 

20. 20. Acosta FJ , McClendon JJ , O’Shaughnessy BA , et al . Morbidity and mortality after spinal deformity surgery in patients 75 years and older: complications and predictive factors. J Neurosurg Spine. 2011;15(6):667–674. doi:10.3171/2011.7.SPINE10640 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.3171/2011.7.SPINE10640&link_type=DOI) 
    
    [PubMed](https://www.ijssurgery.com/lookup/external-ref?access_num=21888481&link_type=MED&atom=%2Fijss%2F15%2F6%2F1210.atom) 

21. 21. Goz V , Weinreb JH , McCarthy I , Schwab F , Lafage V , Errico TJ . Perioperative complications and mortality after spinal fusions: analysis of trends and risk factors. Spine. 2013;38(22):1970–1976. doi:10.1097/BRS.0b013e3182a62527 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1097/BRS.0b013e3182a62527&link_type=DOI) 

22. 22. Karhade AV , Cha TD , Fogel HA , et al . Predicting prolonged opioid prescriptions in opioid-naïve lumbar spine surgery patients. The Spine Journal. 2020;20(6):888–895. doi:10.1016/j.spinee.2019.12.019 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1016/j.spinee.2019.12.019&link_type=DOI) 

23. 23. Sørensen MS , Gerds TA , Hindsø K , Petersen MM . External validation and optimization of the SPRING model for prediction of survival after surgical treatment of bone metastases of the extremities. Clin Orthop Relat Res. 2018;476(8):1591–1599. doi:10.1097/01.blo.0000534678.44152.ee 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1097/01.blo.0000534678.44152.ee&link_type=DOI) 

24. 24. Mosher ZA , Brabston EW , Ewing MA , et al . Predictive accuracy of the ACS-NSQIP surgical risk calculator for pathologic humerus fracture fixation. J Surg Orthop Adv. 2019;28(4):250–256 
    
    

25. 25. Broda A , Sanford Z , Turcotte J , Patton C . Development of a risk prediction model with improved clinical utility in elective cervical and lumbar spine surgery. Spine. 2020;45(9):E542–E551. doi:10.1097/BRS.0000000000003317 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1097/BRS.0000000000003317&link_type=DOI) 

26. 26. McCarthy MH , Singh P , Maslak J , et al . Can the American College of surgeons risk calculator predict 30-day complications after cervical spine surgery? Clin Spine Surg. 2019;32(9):357–362. doi:10.1097/BSD.0000000000000890 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1097/BSD.0000000000000890&link_type=DOI) 

27. 27. McCarthy MH , Singh P , Maslak J , et al . Can the American College of Surgeons Risk calculator predict 30-day complications after cervical spine surgery? Clin Spine Surg. 2019;32(9):357–362. doi:10.1097/BSD.0000000000000890 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1097/BSD.0000000000000890&link_type=DOI) 

28. 28. Alluri RK , Leland H , Heckmann N . Surgical research using national databases. Ann Transl Med. 2016;4(20):393. doi:10.21037/atm.2016.10.49 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.21037/atm.2016.10.49&link_type=DOI) 

29. 29. De la Garza Ramos R , Nakhla J , Echt M , et al . Risk factors for 30-day readmissions and reoperations after 3-column osteotomy for spinal deformity. Global Spine J. 2018;8(5):483–489. doi:10.1177/2192568217739886 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1177/2192568217739886&link_type=DOI) 

30. 30. Cho SK , Bridwell KH , Lenke LG , et al . Major complications in revision adult deformity surgery: risk factors and clinical outcomes with 2- to 7-year follow-up. Spine. 2012;37(6):489–500. doi:10.1097/BRS.0b013e3182217ab5 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1097/BRS.0b013e3182217ab5&link_type=DOI) 
    
    [PubMed](https://www.ijssurgery.com/lookup/external-ref?access_num=21587110&link_type=MED&atom=%2Fijss%2F15%2F6%2F1210.atom) 

31. 31. Glassman SD , Hamill CL , Bridwell KH , Schwab FJ , Dimar JR , Lowe TG . The impact of perioperative complications on clinical outcome in adult deformity surgery. Spine. 2007;32(24):2764–2770. doi:10.1097/BRS.0b013e31815a7644 
    
    [CrossRef](https://www.ijssurgery.com/lookup/external-ref?access_num=10.1097/BRS.0b013e31815a7644&link_type=DOI) 
    
    [PubMed](https://www.ijssurgery.com/lookup/external-ref?access_num=18007258&link_type=MED&atom=%2Fijss%2F15%2F6%2F1210.atom)