Performance of Three Prognostic Models in Critically Ill Patients with Cancer: A Prospective Study

Background: The aim of the study was to evaluate the performance of “Acute Physiology and Chronic Health Evaluation II” (APACHE-II), “Simplified Acute Physiology Score 3” (SAPS-3), and “APACHE-II Score for Critically Ill Patients with a Solid Tumor” (APACHE-IICCP) models in cancer patients admitted to ICU. Methods: Prospective cohort study of 414 patients with an active solid tumor. Discrimination was assessed by area under receiver operating characteristic (AROC) curves and calibration by Hosmer–Lemeshow goodness-of-fit C test (H–L). Results: The hospital mortality rate was 32.6%. In the total cohort, discrimination was superior for APACHE-IICCP model (AROC 0.98) compared to APACHE-II (AROC 0.96), SAPS-3 (AROC 0.91), and SAPS-3 for Central and South American countries (SAPS-3CSA) (AROC 0.95) models. Calibration was good (p-valueof H–L test > 0.05) using APACHE-IICCP, APACHE-II and SAPS-3CSA models. Estimation of the probability of death was more precise with APACHE-IICCP model (standardized mortality ratio, SMR = 1.03). Further analysis showed that discrimination was better with APACHE-IICCP model than with APACHE-II, SAPS-3, and SAPS-3CSA models whether for patients with planned ICU admission (AROC 0.97 vs. 0.96, 0.95 and 0.95, respectively) or for patients with unplanned ICU admission (AROC 0.97 vs. 0.94, 0.86 and 0.95). When the SMR and calibration were analyzed, the APACHE-IICCP model was the only model to provide predicted mortality rates closer to the observed mortality for patients with planned and for patients with unplanned ICU admission. Conclusions: In this prospective study, APACHE-IICCP model was superior to APACHE-II, SAPS-3, and SAPS3CSA models in predicting hospital mortality.


Introduction
Predicting mortality is one of the greatest challenges for critical care physicians. Severity scoring systems integrate clinical data to estimate the probability of death, which can be used to facilitate resource utilization or continuing quality improvement and to stratify patients for clinical research [1].
Prospective validation in different populations of critically ill patients improves the applicability of the model to these settings. Furthermore, the performance of a prognostic score must be evaluated prior to being used in a particular subgroup of patients. Cancer patients account for 13-18% of patients admitted to intensive care units (ICU) [2,3]. General predictive models have been evaluated in cancer patients admitted to ICU [4,5]; however, generalization is difficult because of the limitations of studies.
Groeger, et al. [6] developed a specific prognostic model for critically ill cancer patients two decade ago, but it is not superior to general predictive models [4]. Recently, Martos-Benítez, et al. [7] developed the "APACHE-II Score for Critically Ill Patients with a Solid Tumor" (APACHE-II CCP ) model. This is a specific prognostic model based on a general predictive model, tween groups was assessed using Pearson´s chi-square test (χ 2 ) for categorical variables; the Mann-Whitney U test was used for numerical variables.
Statistical analyses were performed in the same way as Soares, et al. [4]. Discrimination was evaluated by calculating the area under receiver operating characteristic curve (AROC); an AROC of 1.0 denotes perfect discrimination, while a value close to 0.50 indicates no apparent accuracy. The Hosmer-Lemeshow goodness-of-fit C test (H -L test) was used to evaluate the calibration. A p-value > 0.05 indicates a good fit for the model. Calibration curves were constructed by plotting predicted mortality rates stratified by 10% intervals of mortality risk (x-axis) against observed mortality rates (y-axis) using Microsoft Excel software. Standardized mortality rate (SMR) with respective 95% confidence intervals (CI) was calculated for each model by dividing observed by predicted mortality rate.
Two analyses of models' performance were carried out. In the first, all patients were evaluated. In the second, a subgroup analysis (for patients with planned ICU admission and for patients with unplanned ICU admission) was done.
Hypothesis tests showing a two-tailed p-value of < 0.05 were considered as significant. IBM® SPSS® 21.0 (IBM, Armonk, NY, USA) was used for all analyses.
At ICU admission, the median SOFA score was 2.0 points (IQR 1.0-4.0 points), SAPS-3 score was 52.5 points (IQR 46.0-67.0 points), APACHE-II score was 10.0 points (IQR 7.0-15.0 points), and APACHE-II CCP (component 2) score was 1.0 points (IQR 0.0-5.0 points). Invasive mechanical ventilation within 24 hours of ICU admission was used in 144 patients (34.8%). During the ICU stay, MV was required in 193 patients (46.6%). Vasopressors which is believed as better adapted to critically ill patients with cancer. The aim of the present study was to evaluate the performance of "Acute Physiology and Chronic Health Evaluation II" (APACHE-II), "Simplified Acute Physiology Score 3" (SAPS-3), and APACHE-II CCP models in patients with cancer admitted to ICU.

Design and setting
This was a prospective study conducted in the medical and surgical ICU-8B of the "Hermanos Ameijeiras" Hospital between July 2016 and September 2018. This is a 640-bed, university-affiliated, tertiary referral general hospital in Havana, Cuba. The ICU-8B has 12 beds and provides care for approximately 450 critically ill patients per year. The current study was conducted in accordance with the Declaration of Helsinki. It was approved by the Scientific Council and the Ethics Committee for Scientific Research of the center (Number CE 2016/42). Written informed consent was obtained from all included patients.

Participants
During the study period, 482 patients with a solid tumor aged ≥ 19-years-old requiring ICU admission were evaluated. Patients in complete remission for more than two years (9 patients) and those with an ICU stay < 24 hours were excluded (59 patients) (Electronic Supplementary Material 1A). Finally, 414 patients were analyzed.

Data collection and definitions
Demographic, clinical and laboratory variables required to calculate the models were collected at ICU admission (SAPS-3) and during the first 24 hours of ICU admission (APACHE-II and APACHE-II CCP ). Predicted mortality rates were calculated as recommended for each predictive model [7][8][9] (Electronic Supplementary Material 2).
The following variables were also recorded: reason for ICU admission, primary tumor location, clinical-stage of cancer, type of admission (planned ICU-admission and unplanned ICU-admission), nature of admission (malignancy-related or non-malignancy-related) [10], neutropenia (neutrophil count < 500/mm 3 ), organsupport therapy (e.g. vasopressor, invasive mechanical ventilation (MV), renal replacement therapy). Invasive mechanical ventilation ≥ 24 hours was required in postoperative patients. For sedated patients, the Glasgow Coma Scale prior to sedation was used. Sepsis was diagnosed according to the current definitions [11]. The hospital mortality was the main end point.

Statistical analyses
Categorical variables are showed as absolute numbers with percentage, and numerical variables as median with 25 th -75 th interquartile range (IQR). Difference be-unplanned ICU admission than those for patients with planned ICU admission (unplanned 68.8% vs. planned 13.3%; p < 0.0001) ( Table 1).

Performance of predictive models
Discrimination was very good for all models in the total cohort. AROC of APACHE-II CCP model (0.98) was greater than those observed for APACHE-II (0.96), SPAS-3 (0.91), and SAPS-3 for Central and South American countries (SAPS-3 CSA ) (0.95) models (Table 2 and Figure  1). Calibration was good for all models (all p-value > 0.05) except for SAPS-3 model (Table 2 and Figure 2). In term of SMR, APACHE-II CCP and SAPS-3 models were better in predicting the hospital mortality. APACHE-II CCP , APACHE-II and SAPS-3 models underestimated the hospital mortality (SMR > 1.0), while SAPS-3 CSA model overestimated the hospital mortality (SMR < 1.0) ( Table  2). Nevertheless, when the SMR and calibration curves were analyzed, the APACHE-II CCP was the only model to provide predicted mortality rates closer to the observed mortality (Table 2 and Figure 2).
The performance of the models was further analyzed for patients with planned ICU admission and for patients with unplanned ICU admission. The results are presented in Table 3. Discrimination power was good for all predictive models whether for patients with planned ICU admission or for patients with unplanned ICU admission, but again it was better with APACHE-II CCP model than with APACHE-II, SAPS-3 and SAPS-3 CSA models. AROC of APACHE-II CCP model (0.97) was greater than those observed for APACHE-II (0.96), SAPS-3 (0.95), and SAPS-3 CSA (0.95) models in patients with planned ICU admission. AROC of APACHE-II CCP model (0.97) was greater than those observed for APACHE-II (0.94), SAPS-3 (0.86), and SAPS-3 CSA (0.95) models in patients with unplanned ICU admission (Table 3     bration curves were analyzed, the APACHE-II CCP was the only model to provide predicted mortality rates closer to the observed mortality for patients with planned and for those with unplanned ICU admission (Table 3 and Electronic Supplementary Material 1C-1D).

Discussion
Prognostic models are used as surrogate markers of severity of acute illness; however, they must be validated prior to being used in a specific setting or plementary Material 1C). In the subgroup of patients with unplanned ICU admission, all predictive models showed good calibration (all p-value > 0.05) ( Table 3 and Electronic Supplementary Material 1D). All models overestimated the hospital mortality (SMR < 1.0) in patients with planned ICU admission. APACHE-II CCP and SAPS-3 CSA models were accurate in predicting the hospital mortality for patients with unplanned ICU admission, while APACHE-II and SAPS-3 models underestimated the hospital mortality (SMR > 1.0). When the SMR and cali-    health status of patients (assessed by APACHE-II score), takes into account an organ-support method with a powerful association with mortality in critically ill cancer patients (e.g. invasive mechanical ventilation) [19,20], and comprises two cancer-specific variables (e.g. clinical stage of cancer and non-malignancy-related admission) related with mortality in cancer patients admitted to ICU [10,21]. Thus, it is not surprising a better fitting and discrimination with APACHE-II CCP model than those observed with general predictive models.
With the introduction of the new models (e.g. SAPS-3 and APACHE-IV), it has been suggested that the older models should no longer be used because they become increasingly inaccurate. New scoring systems could perform better than older models (e.g. APACHE-II) due to the introduction of more predictive variables [22]. However, the APACHE-II model is still widely used all over the world by four reasons [23,24]: 1-) APACHE-II model is simpler compared with APACHE-IV model; 2-) APACHE-IV model is difficult to be implemented in developing countries because the burden of manual data collection of a lot of variables; 3-) The computing program for running APACHE-IV model is not free of cost; and 4-) Data collection within 1-hour of ICU admission is required for calculating SAPS-3 model, which is not always feasible. The APACHE-II CCP model is as simple as the APACHE-II model so it may be successfully used on all medical scenarios.
The APACHE-II CCP model may be used by clinicians, researchers and administrators in the field of critical care to characterize cancer patients in terms of severity of illness in clinical studies or for the evaluation of ICU performance. Other advantages of APACHE-II CCP model include prediction of cancer patient outcomes, evaluating the efficiency of treatments, using as an efficient in-hospital triage, and measuring the quality improvement of treatment and preventive plans in these patients.
The present study has several shortcomings. Firstly, this was a single-center study. Consequently, differences in bed availability among ICUs, patient's composition and standard of care are likely to have an adverse impact on predictive accuracy in other studies. Secondly, although the sample size was adequate considering the statistical power of the study, validation of prognostic models in critically ill patients with cancer is desired in a large cohort. Thirdly, an individual mortality prediction contains an error. Hence, a prediction is a punctual estimate of an individual's probability of death, where confidence interval needs to be considered.
In this prospective cohort, we found that APACHE-II CCP model was an accurate scoring system for predicting mortality in patients with cancer admitted to ICU. Further prospective studies confirming our results are required.
in a subgroup of patients, such as critically ill patients with cancer. In fact, prognostic models have not been sufficiently evaluated in this subpopulation.
Two general prognostic models (APACHE-II and SAPS-3) and one cancer-specific prognostic model were evaluated in this study. APACHE-II, SAPS-3, and APACHE-II CCP models were chosen for comparisons because these models are currently used in our ICU and are one of the most popular models around the world [12][13][14]. According to the study objective, we found a better performance for APACHE-II CCP model than those for APACHE-II, SAPS-3, and SAPS-3 CSA models.
Heterogeneity of the study population is almost the rule in any study examining critical ill patients with cancer because of the differences in the purpose of treatment (medical or surgical), variety in clinical course of disease (clinical stage, progression or recurrence), diversity in histological pattern, and different primary tumor sites. The acute clinical condition is another factor to take into account. Thus, a prognostic model involving all these topics is a chimera. The major purpose is to develop a model with the best possible accuracy for predicting mortality in this subpopulation.
In critically ill cancer patients, Soares, et al. [4] observed that discrimination was good using SAPS-3 and SAPS-3 CSA models. Recently, Xing, et al. [5] validated APACHE-II, APACHE-IV, and SAPS-3 models. Authors found that the performance of the three models was good. All predictive models showed an excellent discrimination (AROC > 0.90 for all models) and a good calibration in the present prospective cohort. Conversely, in the APACHE-II CCP model developing study, the performance of APACHE-II model (AROC = 0.62) was really poor compared with APACHE-II CCP model (AROC = 0.91) [7]. Although APACHE-II CCP model seem to be better for predicting mortality in critically ill patients with cancer, general models should be not discharged until reliable studies in different setting.
The survival rates for patients with unplanned ICU admission are different than those for patients with planned ICU admission [2,[15][16][17], in part because elective surgery is the most common reason for admission to ICU in cancer patients [18]. In view of our results, APACHE-II CCP model is accurate for predicting mortality in critically ill patients with cancer whether in those with planned ICU admission or unplanned ICU admission. However, a customized equation for type of admission is desired and must be developed in future research.
The improved performance obtained with APACHE-II CCP model is explained, at least in part, by two reasons. First, the APACHE-II CCP model is a cancer-specific predictive model built in cancer patients admitted to ICU rather than in a general population of critically ill patients. Second, the APACHE-II CCP model measures the acute pathophysiology imbalance and the chronic • Page 9 of 11 •