The aim of the study was to evaluate the performance of "Acute Physiology and Chronic Health Evaluation II" (APACHE-II), "Simplified Acute Physiology Score 3" (SAPS-3), and "APACHE-II Score for Critically Ill Patients with a Solid Tumor" (APACHE-IICCP) models in cancer patients admitted to ICU.
Prospective cohort study of 414 patients with an active solid tumor. Discrimination was assessed by area under receiver operating characteristic (AROC) curves and calibration by Hosmer-Lemeshow goodness-of-fit C test (H-L).
The hospital mortality rate was 32.6%. In the total cohort, discrimination was superior for APACHE-IICCP model (AROC 0.98) compared to APACHE-II (AROC 0.96), SAPS-3 (AROC 0.91), and SAPS-3 for Central and South American countries (SAPS-3CSA) (AROC 0.95) models. Calibration was good (p-valueof H-L test > 0.05) using APACHE-IICCP, APACHE-II and SAPS-3CSA models. Estimation of the probability of death was more precise with APACHE-IICCP model (standardized mortality ratio, SMR = 1.03). Further analysis showed that discrimination was better with APACHE-IICCP model than with APACHE-II, SAPS-3, and SAPS-3CSA models whether for patients with planned ICU admission (AROC 0.97 vs. 0.96, 0.95 and 0.95, respectively) or for patients with unplanned ICU admission (AROC 0.97 vs. 0.94, 0.86 and 0.95). When the SMR and calibration were analyzed, the APACHE-IICCP model was the only model to provide predicted mortality rates closer to the observed mortality for patients with planned and for patients with unplanned ICU admission.
In this prospective study, APACHE-IICCP model was superior to APACHE-II, SAPS-3, and SAPS-3CSA models in predicting hospital mortality.
APACHE, Cancer, Intensive care unit, Mortality, Prognostic score, SAPS
Predicting mortality is one of the greatest challenges for critical care physicians. Severity scoring systems integrate clinical data to estimate the probability of death, which can be used to facilitate resource utilization or continuing quality improvement and to stratify patients for clinical research [1].
Prospective validation in different populations of critically ill patients improves the applicability of the model to these settings. Furthermore, the performance of a prognostic score must be evaluated prior to being used in a particular subgroup of patients. Cancer patients account for 13-18% of patients admitted to intensive care units (ICU) [2,3]. General predictive models have been evaluated in cancer patients admitted to ICU [4,5]; however, generalization is difficult because of the limitations of studies.
Groeger, et al. [6] developed a specific prognostic model for critically ill cancer patients two decade ago, but it is not superior to general predictive models [4]. Recently, Martos-Benítez, et al. [7] developed the "APACHE-II Score for Critically Ill Patients with a Solid Tumor" (APACHE-IICCP) model. This is a specific prognostic model based on a general predictive model, which is believed as better adapted to critically ill patients with cancer. The aim of the present study was to evaluate the performance of "Acute Physiology and Chronic Health Evaluation II" (APACHE-II), "Simplified Acute Physiology Score 3" (SAPS-3), and APACHE-IICCP models in patients with cancer admitted to ICU.
This was a prospective study conducted in the medical and surgical ICU-8B of the "Hermanos Ameijeiras" Hospital between July 2016 and September 2018. This is a 640-bed, university-affiliated, tertiary referral general hospital in Havana, Cuba. The ICU-8B has 12 beds and provides care for approximately 450 critically ill patients per year. The current study was conducted in accordance with the Declaration of Helsinki. It was approved by the Scientific Council and the Ethics Committee for Scientific Research of the center (Number CE 2016/42). Written informed consent was obtained from all included patients.
During the study period, 482 patients with a solid tumor aged ≥ 19-years-old requiring ICU admission were evaluated. Patients in complete remission for more than two years (9 patients) and those with an ICU stay < 24 hours were excluded (59 patients) (Electronic Supplementary Material 1A). Finally, 414 patients were analyzed.
Demographic, clinical and laboratory variables required to calculate the models were collected at ICU admission (SAPS-3) and during the first 24 hours of ICU admission (APACHE-II and APACHE-IICCP). Predicted mortality rates were calculated as recommended for each predictive model [7-9] (Electronic Supplementary Material 2).
The following variables were also recorded: reason for ICU admission, primary tumor location, clinical-stage of cancer, type of admission (planned ICU-admission and unplanned ICU-admission), nature of admission (malignancy-related or non-malignancy-related) [10], neutropenia (neutrophil count < 500/mm3), organ-support therapy (e.g. vasopressor, invasive mechanical ventilation (MV), renal replacement therapy). Invasive mechanical ventilation ≥ 24 hours was required in postoperative patients. For sedated patients, the Glasgow Coma Scale prior to sedation was used. Sepsis was diagnosed according to the current definitions [11]. The hospital mortality was the main end point.
Categorical variables are showed as absolute numbers with percentage, and numerical variables as median with 25th - 75th interquartile range (IQR). Difference between groups was assessed using Pearson's chi-square test (χ2) for categorical variables; the Mann-Whitney U test was used for numerical variables.
Statistical analyses were performed in the same way as Soares, et al. [4]. Discrimination was evaluated by calculating the area under receiver operating characteristic curve (AROC); an AROC of 1.0 denotes perfect discrimination, while a value close to 0.50 indicates no apparent accuracy. The Hosmer-Lemeshow goodness-of-fit C test (H - L test) was used to evaluate the calibration. A p-value > 0.05 indicates a good fit for the model. Calibration curves were constructed by plotting predicted mortality rates stratified by 10% intervals of mortality risk (x-axis) against observed mortality rates (y-axis) using Microsoft Excel software. Standardized mortality rate (SMR) with respective 95% confidence intervals (CI) was calculated for each model by dividing observed by predicted mortality rate.
Two analyses of models' performance were carried out. In the first, all patients were evaluated. In the second, a subgroup analysis (for patients with planned ICU admission and for patients with unplanned ICU admission) was done.
Hypothesis tests showing a two-tailed p-value of < 0.05 were considered as significant. IBM® SPSS® 21.0 (IBM, Armonk, NY, USA) was used for all analyses.
The most common primary tumor sites were pancreas/liver/biliary tract (23.9%), lower gastrointestinal (22.7%), upper gastrointestinal (14.5%) and intracranial (10.4%). Of the hematological malignancies (5.6%), 17 (73.9%) were non-Hodgkin's lymphomas and 6 (26.1%) were Hodgkin's lymphomas. The median time from cancer diagnosis to inclusion of patients in the study was 14.0 months (IQR 6.0-20.0 months). Comorbidities were reported in 332 (80.2%) patients; the most frequent being arterial hypertension (54.5%), diabetes mellitus (29.2%), chronic pulmonary disease (18.4%), and coronary artery disease (7.8%).
Reasons for ICU admission were scheduled postoperative care (65.2%), sepsis/septic shock (18.4%), primary respiratory failure (14.5%), and others (1.9%). Patients were admitted from operating/recovery rooms (70.5%), ward/floor (22.7%), emergency department (4.3%) and other ICUs (2.6%).
At ICU admission, the median SOFA score was 2.0 points (IQR 1.0-4.0 points), SAPS-3 score was 52.5 points (IQR 46.0-67.0 points), APACHE-II score was 10.0 points (IQR 7.0-15.0 points), and APACHE-IICCP (component 2) score was 1.0 points (IQR 0.0-5.0 points). Invasive mechanical ventilation within 24 hours of ICU admission was used in 144 patients (34.8%). During the ICU stay, MV was required in 193 patients (46.6%). Vasopressors on ICU admission were used in 68 patients (16.4%), and 102 patients (24.6%) required vasopressors through the ICU stay. Patients needing renal replacement therapy were 16 (3.9%). Patients' characteristics on ICU admission are displayed in Table 1.
Table 1: Patients' characteristics on ICU admission and outcomes. View Table 1
The median in length of ICU stay (non-survivors 6.0 days [IQR 5.0-10.0 days] vs. survivors 4.0 days [IQR 2.0-6.0 days]; p < 0.0001) and length of hospital stay (non-survivors 18.0 days [IQR 17.0-22.0 days] vs. survivors 17.0 days [IQR 15.0-19.0 days]; p < 0.0001) were higher in non-survivors than in survivors. The ICU and hospital mortality rates were 24.9% and 32.6%, respectively. Hospital mortality rate was higher for patients with unplanned ICU admission than those for patients with planned ICU admission (unplanned 68.8% vs. planned 13.3%; p < 0.0001) (Table 1).
The SAPS-3, APACHE-II and APACHE-IICCP (component 2) scores were higher for patients with unplanned ICU admission (68.5 points [IQR 53.0-78.0 points], 14.0 points [IQR 11.0-19.0 points] and 5.0 points [IQR 2.0-6.0 points], respectively) than for those with planned ICU admission (49.0 points [IQR 44.0-56.0 points], 9.0 points [IQR 6.0-12.0 points] and 0.0 points [IQR 0.0-1.0 points], respectively) (p < 0.0001 for all comparisons). As expected, these scores were also higher in non-survivors (SAPS-3 score 73.0 points [IQR 58.0-79.0 points], APACHE-II score 15.0 points [IQR 12.0-19.0 points] and APACHE-IICCP (component 2) score 5.0 points [IQR 5.0-6.0 points]) than in survivors (SAPS-3 score 49.0 points [IQR 44.0-55.0 points], APACHE-II score 9.0 points [IQR 6.0-12.0 points] and APACHE-IICCP (component 2) score 0.0 points [IQR 0.0-1.0 points]) (p < 0.0001 for all comparisons).
Discrimination was very good for all models in the total cohort. AROC of APACHE-IICCP model (0.98) was greater than those observed for APACHE-II (0.96), SPAS-3 (0.91), and SAPS-3 for Central and South American countries (SAPS-3CSA) (0.95) models (Table 2 and Figure 1). Calibration was good for all models (all p-value > 0.05) except for SAPS-3 model (Table 2 and Figure 2). In term of SMR, APACHE-IICCP and SAPS-3 models were better in predicting the hospital mortality. APACHE-IICCP, APACHE-II and SAPS-3 models underestimated the hospital mortality (SMR > 1.0), while SAPS-3CSA model overestimated the hospital mortality (SMR < 1.0) (Table 2). Nevertheless, when the SMR and calibration curves were analyzed, the APACHE-IICCP was the only model to provide predicted mortality rates closer to the observed mortality (Table 2 and Figure 2).
Figure 1: Receiver operating characteristic curve for APACHE-IICCP, APACHE-II, SAPS-3 and SAPS-3CSA models. The diagonal line represents the reference line. APACHE-II, Acute Physiology and Chronic Health Evaluation II; APACHE-IICCP, APACHE-II score for critically ill patients with a solid tumor; SAPS-3, Simplified Acute Physiology Score 3; SAPS-3CSA, the customized equation of SAPS-3 score for Central and South American countries. View Figure 1
Figure 2: Calibration curves for APACHE-IICCP, APACHE-II, SAPS-3 and SAPS-3CSA models. (A) "APACHE-II score for critically ill patients with a solid tumor" model (p = 0.272); (B) "Acute Physiology and Chronic Health Evaluation II" model (p = 0.183); (C) "Simplified Acute Physiology Score 3" model (p = 0.003); and (D) "Simplified Acute Physiology Score 3 for Central and South American countries" model (p = 0.082). View Figure 2
Table 2: Performance for the different prognostic models (n = 414; observed hospital mortality = 32.6%). View Table 2
The performance of the models was further analyzed for patients with planned ICU admission and for patients with unplanned ICU admission. The results are presented in Table 3. Discrimination power was good for all predictive models whether for patients with planned ICU admission or for patients with unplanned ICU admission, but again it was better with APACHE-IICCP model than with APACHE-II, SAPS-3 and SAPS-3CSA models. AROC of APACHE-IICCP model (0.97) was greater than those observed for APACHE-II (0.96), SAPS-3 (0.95), and SAPS-3CSA (0.95) models in patients with planned ICU admission. AROC of APACHE-IICCP model (0.97) was greater than those observed for APACHE-II (0.94), SAPS-3 (0.86), and SAPS-3CSA (0.95) models in patients with unplanned ICU admission (Table 3 and Electronic Supplementary Material 1B). Calibration was good for all models (all p-value > 0.05) except for APACHE-II model in patients with planned ICU admission (Table 3 and Electronic Supplementary Material 1C). In the subgroup of patients with unplanned ICU admission, all predictive models showed good calibration (all p-value > 0.05) (Table 3 and Electronic Supplementary Material 1D). All models overestimated the hospital mortality (SMR < 1.0) in patients with planned ICU admission. APACHE-IICCP and SAPS-3CSA models were accurate in predicting the hospital mortality for patients with unplanned ICU admission, while APACHE-II and SAPS-3 models underestimated the hospital mortality (SMR > 1.0). When the SMR and calibration curves were analyzed, the APACHE-IICCP was the only model to provide predicted mortality rates closer to the observed mortality for patients with planned and for those with unplanned ICU admission (Table 3 and Electronic Supplementary Material 1C-1D).
Table 3: Performance for the different prognostic models in patients with planned and unplanned ICU admission. View Table 3
Prognostic models are used as surrogate markers of severity of acute illness; however, they must be validated prior to being used in a specific setting or in a subgroup of patients, such as critically ill patients with cancer. In fact, prognostic models have not been sufficiently evaluated in this subpopulation.
Two general prognostic models (APACHE-II and SAPS-3) and one cancer-specific prognostic model were evaluated in this study. APACHE-II, SAPS-3, and APACHE-IICCP models were chosen for comparisons because these models are currently used in our ICU and are one of the most popular models around the world [12-14]. According to the study objective, we found a better performance for APACHE-IICCP model than those for APACHE-II, SAPS-3, and SAPS-3CSA models.
Heterogeneity of the study population is almost the rule in any study examining critical ill patients with cancer because of the differences in the purpose of treatment (medical or surgical), variety in clinical course of disease (clinical stage, progression or recurrence), diversity in histological pattern, and different primary tumor sites. The acute clinical condition is another factor to take into account. Thus, a prognostic model involving all these topics is a chimera. The major purpose is to develop a model with the best possible accuracy for predicting mortality in this subpopulation.
In critically ill cancer patients, Soares, et al. [4] observed that discrimination was good using SAPS-3 and SAPS-3CSA models. Recently, Xing, et al. [5] validated APACHE-II, APACHE-IV, and SAPS-3 models. Authors found that the performance of the three models was good. All predictive models showed an excellent discrimination (AROC > 0.90 for all models) and a good calibration in the present prospective cohort. Conversely, in the APACHE-IICCP model developing study, the performance of APACHE-II model (AROC = 0.62) was really poor compared with APACHE-IICCP model (AROC = 0.91) [7]. Although APACHE-IICCP model seem to be better for predicting mortality in critically ill patients with cancer, general models should be not discharged until reliable studies in different setting.
The survival rates for patients with unplanned ICU admission are different than those for patients with planned ICU admission [2,15-17], in part because elective surgery is the most common reason for admission to ICU in cancer patients [18]. In view of our results, APACHE-IICCP model is accurate for predicting mortality in critically ill patients with cancer whether in those with planned ICU admission or unplanned ICU admission. However, a customized equation for type of admission is desired and must be developed in future research.
The improved performance obtained with APACHE-IICCP model is explained, at least in part, by two reasons. First, the APACHE-IICCP model is a cancer-specific predictive model built in cancer patients admitted to ICU rather than in a general population of critically ill patients. Second, the APACHE-IICCP model measures the acute pathophysiology imbalance and the chronic health status of patients (assessed by APACHE-II score), takes into account an organ-support method with a powerful association with mortality in critically ill cancer patients (e.g. invasive mechanical ventilation) [19,20], and comprises two cancer-specific variables (e.g. clinical stage of cancer and non-malignancy-related admission) related with mortality in cancer patients admitted to ICU [10,21]. Thus, it is not surprising a better fitting and discrimination with APACHE-IICCP model than those observed with general predictive models.
With the introduction of the new models (e.g. SAPS-3 and APACHE-IV), it has been suggested that the older models should no longer be used because they become increasingly inaccurate. New scoring systems could perform better than older models (e.g. APACHE-II) due to the introduction of more predictive variables [22]. However, the APACHE-II model is still widely used all over the world by four reasons [23,24]: 1-) APACHE-II model is simpler compared with APACHE-IV model; 2-) APACHE-IV model is difficult to be implemented in developing countries because the burden of manual data collection of a lot of variables; 3-) The computing program for running APACHE-IV model is not free of cost; and 4-) Data collection within 1-hour of ICU admission is required for calculating SAPS-3 model, which is not always feasible. The APACHE-IICCP model is as simple as the APACHE-II model so it may be successfully used on all medical scenarios.
The APACHE-IICCP model may be used by clinicians, researchers and administrators in the field of critical care to characterize cancer patients in terms of severity of illness in clinical studies or for the evaluation of ICU performance. Other advantages of APACHE-IICCP model include prediction of cancer patient outcomes, evaluating the efficiency of treatments, using as an efficient in-hospital triage, and measuring the quality improvement of treatment and preventive plans in these patients.
The present study has several shortcomings. Firstly, this was a single-center study. Consequently, differences in bed availability among ICUs, patient's composition and standard of care are likely to have an adverse impact on predictive accuracy in other studies. Secondly, although the sample size was adequate considering the statistical power of the study, validation of prognostic models in critically ill patients with cancer is desired in a large cohort. Thirdly, an individual mortality prediction contains an error. Hence, a prediction is a punctual estimate of an individual's probability of death, where confidence interval needs to be considered.
In this prospective cohort, we found that APACHE-IICCP model was an accurate scoring system for predicting mortality in patients with cancer admitted to ICU. Further prospective studies confirming our results are required.
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
The authors declare that they have no competing interest.
FDMB: concepts, design, definition of intellectual content, literature search, clinical studies, data acquisition, data analysis, statistical analysis, manuscript preparation, manuscript editing, and manuscript review; HLM y DLP: design, definition of intellectual content, clinical studies, data acquisition, data analysis, manuscript editing, and manuscript review; JCRL, VOR y JLMA: literature search, data acquisition, data analysis, manuscript preparation, manuscript editing, and manuscript review.