Vol. 43 No.

3 March 2012 Journal of Pain and Symptom Management 527

Original Article

Development of a Prognostic Model for

Six-Month Mortality in Older Adults With
Declining Health
Paul K.J. Han, MD, MA, MPH, Minjung Lee, PhD, Bryce B. Reeve, PhD,
Angela B. Mariotto, PhD, Zhuoqiao Wang, MS, Ron D. Hays, PhD,
K. Robin Yabroff, PhD, Marie Topor, MS, and Eric J. Feuer, PhD
Center for Outcomes Research and Evaluation (P.K.J.H.), Maine Medical Center, Portland, Maine;
Division of Cancer Control and Population Sciences (M.L., B.B.R., A.B.M., K.R.Y., E.J.F.), National
Cancer Institute, Bethesda, Maryland; Information Management Services, Inc. (Z.W., M.T.),
Silver Spring, Maryland; and Division of General Internal Medicine (R.D.H.), University of
California at Los Angeles, Los Angeles, California, USA

Abstract
Context. Estimation of six-month prognosis is essential in hospice referral
decisions, but accurate, evidence-based tools to assist in this task are lacking.
Objectives. To develop a new prognostic model, the Patient-Reported Outcome
Mortality Prediction Tool (PROMPT), for six-month mortality in community-
dwelling elderly patients.
Methods. We used data from the Medicare Health Outcomes Survey linked to
vital status information. Respondents were 65 years old or older, with self-reported
declining health over the past year (n ¼ 21,870), identified from four Medicare
Health Outcomes Survey cohorts (1998e2000, 1999e2001, 2000e2002, and
2001e2003). A logistic regression model was derived to predict six-month
mortality, using sociodemographic characteristics, comorbidities, and health-
related quality of life (HRQOL), ascertained by measures of activities of daily
living and the Medical Outcomes Study Short Form-36 Health Survey; k-fold cross-
validation was used to evaluate model performance, which was compared with
existing prognostic tools.
Results. The PROMPT incorporated 11 variables, including four HRQOL
domains: general health perceptions, activities of daily living, social functioning,
and energy/fatigue. The model demonstrated good discrimination
(c-statistic ¼ 0.75) and calibration. Overall diagnostic accuracy was superior to
existing tools. At cut points of 10%e70%, estimated six-month mortality risk
sensitivity and specificity ranged from 0.8% to 83.4% and 51.1% to 99.9%,
respectively, and positive likelihood ratios at all mortality risk cut points $40%
exceeded 5.0. Corresponding positive and negative predictive values were 23.1%
e64.1% and 85.3%e94.5%. Over 50% of patients with estimated six-month
mortality risk $30% died within 12 months.

Adddress correspondence to: Paul K.J. Han, MD, MA, Accepted for publication: April 20, 2011.
MPH, Center for Outcomes Research and Evaluation,
Maine Medical Center, 39 Forest Avenue, Portland,
ME 04105, USA. E-mail: hanp@mmc.org

Ó 2012 U.S. Cancer Pain Relief Committee 0885-3924/$ - see front matter
Published by Elsevier Inc. All rights reserved. doi:10.1016/j.jpainsymman.2011.04.015
528 Han et al. Vol. 43 No. 3 March 2012

Conclusion. The PROMPT, a new prognostic model incorporating HRQOL,

demonstrates promising performance and potential value for hospice referral
decisions. More work is needed to evaluate the model. J Pain Symptom Manage
2012;43:527e539. Ó 2012 U.S. Cancer Pain Relief Committee. Published by Elsevier Inc.
All rights reserved.

Key Words
Prognosis, clinical prediction model, elderly, health-related quality of life

Introduction rigorously evaluated model, from the Study to

Understand Prognoses and Preferences for Out-
Prognostic estimates are important in nu-
comes and Risks of Treatments (SUPPORT),
merous medical decisions, but perhaps no-
used disease characteristics and physiologic vari-
where do they play a more critical role than
ables to predict six-month mortality in critically
in the decision to initiate hospice care. This
ill patients surviving hospitalization for any of
single decision formalizes the beginning of
nine serious illnesses.21 However, in a subse-
the end-of-life period and the transition from
quent validation effort in patients with advanced
curative to palliative goals of care for many
lung, heart, and liver disease and a 25% six-
patients, and thus implicitly embodies some
month mortality, this model also demonstrated
estimate of prognosis. In the U.S., the decision
poor performance.19
to initiate hospice care explicitly depends on
More recent modeling efforts are promising
prognostic estimates because physicians must
but have been limited to specific diseases,
certify an expected survival of six months or
such as cancer and dementia,22e26 or temporal
less before patients can receive services under
endpoints other than six months. For example,
the Medicare Hospice Benefit.1,2
models have been developed to predict short-
It follows that difficulties in accurately and
term (less than six months) mortality27,28 in
prospectively estimating six-month mortality
terminally ill patients already referred for
may be an important determinant of the under-
hospice or palliative care and long-term (one
utilization of hospice servicesdand the propor-
year or more) mortality in hospitalized29 or
tionate overutilization of aggressive curative
community-dwelling elders.30,31 These models
interventionsdknown to characterize end-of-
are thus less useful for hospice referral deci-
life care in the U.S.3e15 Physicians’ prognostic
sions, although they demonstrate improved
estimates are known to be generally inaccurate
predictive performance, possibly because of
and optimistically biased.14,16,17 The extent
their inclusion of patient-reported outcomes
and systematic nature of this inaccuracy and
(PROs), such as self-reported functioning and
its correspondence with existing patterns of
well-being or health-related quality of life
end-of-life care suggest that prognostic uncer-
(HRQOL). Accumulating evidence suggests
taintydparticularly with respect to six-month
that PROs assume greater prognostic power
mortalitydmay be a key determinant of hospice
than other variables as the end of life ap-
underutilization.
proaches, presumably because the dying pro-
Yet, few accurate and evidence-based prog-
cess represents a final common pathway
nostic tools exist to help clinicians estimate
characterized by a relatively small set of symp-
six-month mortality. Consensus guidelines
toms and functional impairments.14,32e47
were developed in 1996 by the National Hos-
PROs also are attractive as prognostic variables
pice Organization (NHO) and adopted widely
because they are feasibly ascertainable directly
by clinicians and health policymakers.18 How-
from patients.
ever, these guidelines were not evidence based
To our knowledge, however, there have been
and have been shown to perform poorly in pre-
no previous attempts to integrate HRQOL into
dicting six-month mortality.19,20 Empirically
prognostic models for six-month mortality in
derived prognostic models also have shown lim-
general patient populations. Our objective in
ited accuracy. Perhaps the best-known, most
Vol. 43 No. 3 March 2012 Prognostic Model for Six-Month Mortality 529

the present study was to develop a broadly ap- mail, with telephone follow-up and administra-
plicable prognostic model incorporating tion to initial nonresponders.
HRQOL to predict six-month mortality, the We used data from four MHOS cohorts:
Patient-Reported Outcome Mortality Predic- 1998e2000, 1999e2001, 2000e2002, and
tion Tool (PROMPT), and to explore whether 2001e2003. During this time, baseline and
such a model could have sufficient accuracy to follow-up surveys were distributed to a total of
inform hospice referral decisions. 986,530 patients, of whom 912,703 were eligible
for participation (more than 65 years old, man-
aged care enrollees, not deceased). There were
Methods 634,892 total respondents (response rate 70%);
we analyzed the last survey completed by any
Data Source and Sample Population respondent (Fig. 1). We further limited the
This study used data from the Medicare
sample to patients for whom use of a prognostic
Health Outcomes Survey (MHOS), an annual
tool for hospice decision making would be most
nationwide survey of Medicare managed care
clinically appropriate and useful, using the
beneficiaries administered by the Centers for
SF-36 health transition item to select patients
Medicare and Medicaid Services (CMS) since
reporting significantly declining health: ‘‘Com-
1998 (www.hosonline.org).48e50 The MHOS
pared to one year ago, how would you rate your
surveys a random sample of 1000 Medicare
health in general now?’’ This item is not scored
beneficiaries from each managed care plan un-
in any SF-36 scale, and its use as an inclusion cri-
der contract with CMS (between 250 and 320
terion has conceptual validity because physi-
participating plans yearly). Participants com-
cians should be more apt to consider hospice
plete a baseline and a two-year follow-up survey
referrals for patients with substantially declin-
if enrolled in the same plan. Institutionalized
ing health. We included only respondents who
and disabled beneficiaries are included, but
reported that their health was ‘‘much worse’’
patients on Medicare solely because of end-
(n ¼ 21,870); there were 3295 deaths in this
stage renal disease are excluded. The MHOS
group, yielding a substantially higher observed
uses a self-report questionnaire to collect
six-month mortality than in the overall MHOS
data on patient sociodemographic characteris-
sample (15% vs. 2%).
tics, comorbidities, clinical symptoms, and
HRQOL, as measured by activities of daily
living (ADLs) and the Medical Outcomes Measures
Study Short Form-36 Health Survey (SF-36Ò, Sociodemographic characteristics included in
version 1).51 The MHOS is administered by our analysis were self-reported age, sex, race/

986,530 total patients

(4 MHOS cohorts, 1998-2003)

73,827 ineligible 912,703 eligible patients

277,811 non-respondents 634,892 respondents

(response rate - 70%)

613,022 excluded 21,870 final study sample

Respondents with declining health (3%)
(Self-reported health status in past year: “much worse”)

MHOS study sample consisting of four 2-year cohorts (1998-2001), surveyed from 1998-2003 (N=986,530)

Fig. 1. Derivation of the study sample of adults aged 65 or more with declining health, MHOS.
530 Han et al. Vol. 43 No. 3 March 2012

ethnicity, education, and current marital frequency of missing data (11%), and to avoid
status. dropping cases, we created a dummy variable
Comorbidities included several self-reported for nonresponse to this item. Missing data for
diseases: hypertension, coronary artery disease, all other individual variables were less than
congestive heart failure, other heart condi- 8% and handled through multiple imputation
tions, stroke, chronic obstructive pulmonary using the PROC MI and PROC MI-ANALYZE
disease (COPD), diabetes, and cancer. Arthri- functions of SAS software, version 9.1.3 (SAS
tis, inflammatory bowel disease, and sciatica Institute, Inc., Cary, NC). Missing data were
were ascertained but excluded from analyses imputed using a Markov chain Monte Carlo
because they are not leading causes of mortal- method with multiple chains, creating 10
ity in U.S. adults aged 65 years and older.52 imputed ‘‘complete’’ data sets.
Smoking status (current, former, and never) Because of the large number of potential pre-
also was ascertained. dictors, especially those related to HRQOL, we
HRQOL was measured in two ways. Func- made several decisions to facilitate variable se-
tional status was assessed using six ADLs: bath- lection. We incorporated SF-36 scales rather
ing, dressing, eating, getting in or out of than individual items to maximize measure-
chairs, walking, and using the toilet. These ment precision and because scale scores are
items had three response options: ‘‘No, I do normed to the U.S. general population. To fur-
not have difficulty’’/‘‘Yes, I have difficulty’’/‘‘I ther reduce variables in the model and because
am unable to do this activity.’’ We created a sum- an a priori theoretical justification for variable
mary measure of the total number of ADLs for selection is lacking, we applied a backward elim-
which respondents indicated ‘‘unable to do ination strategy with an Akaike’s information
this activity;’’ scores ranged from zero to six, criterion stopping rule53 to a model, including
with higher scores indicating greater functional all predictor variables in each imputed data set.
impairment. HRQOL also was ascertained us- This is equivalent to using a P-value of 0.157
ing the SF-36 version 1,51 a widely used, compre- for a variable with one degree of freedom.54
hensive, generic health status instrument We then applied the majority method, includ-
comprising eight scales: physical functioning ing variables selected in five or more of the 10
(10 items), role limitations because of physical imputed sets.55
health problems (four items), bodily pain Age, sex, race/ethnicity, education, proxy sta-
(two items), general health perceptions (five tus, hypertension, congestive heart failure,
items), energy/fatigue (four items), social stroke, COPD, presence of any cancer, smoking
functioning (two items), role limitations be- status, ADL score, and SF-36 scores for bodily
cause of emotional problems (three items), pain, general health perceptions, emotional
and emotional well-being (five items). Re- well-being, social functioning, and energy/fa-
sponse options ranged from two to six ordinal tigue were selected into a final model. Of the
categories. SF-36 scale scores were normalized continuous variables, age showed a nonlinear
to the general U.S. population on a T-score met- association with six-month mortality and was,
ric (mean ¼ 50, standard deviation [SD] ¼ 10), therefore, modeled as a restricted cubic spline
with higher scores indicating better HRQOL. with the 5%, 35%, 65%, and 95% percentiles
The outcome variable was survival at six of age.
months since the last completed survey for Some variables showed counterintuitive asso-
each respondent. Vital status and date of death ciations with lower six-month mortality in both
were obtained from the CMS Medicare Enroll- univariate and multivariate analyses: non-white
ment Database. Survey completion by proxy race, low education, hypertension, stroke,
also was ascertained. greater bodily pain, and low emotional well-
being. Some of these counterintuitive associa-
Model Development tions have been found in other studies43,56e59
The large number of predictor variables re- and may reflect confounding by unmeasured
sulted in a substantial proportion of subjects variables (e.g., health care access and quality),
with missing data (n ¼ 6154, representing selection biases that could have altered the rel-
28% of the sample). The proxy survey comple- ative influence of competing mortality risks
tion variable had a disproportionately high (e.g., restriction to a managed care sample,
Vol. 43 No. 3 March 2012 Prognostic Model for Six-Month Mortality 531

selection according to self-reported declining $0.2,., $0.7) and comparing the average
health), or the effects of illness adaptation on predicted six-month mortality in each group
participants’ HRQOL ratings.60,61 Counterintu- with the actual proportion of patients who
itive associations potentially diminish the ‘‘sen- died in six months. To assess calibration graph-
sibility’’ or face validity of risk prediction ically across estimated risk strata, we used
models for clinicians, and many modelers a nonparametric method (PROC LOESS;
thus recommend excluding the variables SAS Institute, Inc., Cary, NC) to produce
involved.62e64 Other modelers have addition- a smoothed high-resolution calibration curve
ally excluded race/ethnicity and education with histogram plot.71
both because of confounding of their prognos- We then calculated sensitivity, specificity, pos-
tic significance and to ethical concerns about itive and negative predictive values (PPVs and
the potential for models incorporating these NPVs), and positive and negative likelihood
variables to contribute to health disparities.31,65 ratios (LRsþ and LRs) at different estimated
For these reasons and to maximize parsimony, mortality risk thresholds to compare perfor-
we conducted sensitivity analyses both includ- mance characteristics of the PROMPT with
ing and excluding counterintuitively associated those of the NHO guidelines and SUPPORT
variables in multivariate regression models. model, reported by Fox et al.19 Finally, we
Model fit, discrimination, and calibration generated Kaplan-Meier curves to compare ex-
were similar and, therefore, we excluded these tended survival of respondents across all risk
variables. Regression coefficients and standard strata.
errors for variables in the final model were com-
puted by averaging across the 10 imputed data
sets, following Rubin’s method.66
Results
Statistical Analyses and Model Evaluation The MHOS and general U.S. population sur-
Because the MHOS sample is composed of vival curves were similar (Fig. 2), although the
Medicare managed care beneficiaries whose MHOS sample had slightly better survival than
access to care, health status, and thus mortality the U.S. population; the ages at median sur-
might differ from the general U.S. popula- vival probability were 85 and 83 years, respec-
tion,67 we generated life tables68,69 comparing tively, for the MHOS and U.S. populations.
overall survival of the MHOS sample with that Table 1 shows the distribution of sociodemo-
of the year 2000 general U.S. population to as- graphic characteristics, comorbidities, and
sess representativeness. We also calculated de- HRQOL.
scriptive statistics on sociodemographic and Table 2 shows the 11 variables included in the
health-related characteristics of the sample. PROMPT and their associations with six-month
We used k-fold cross-validation (k ¼ 10) to mortality in the entire study sample. HRQOL
validate the model. Each of the 10 imputed
data sets was randomly partitioned into k sub-
samples, with each subsample used once as
the validation set and the remaining k  1 set
used as the training set. We assessed model dis-
crimination by calculating the c-statistic or area
under the receiver operating characteristic
curve, averaging the c-statistics across all im-
puted data sets and cross-validation samples.70
To further evaluate prognostic performance of
the final model, we computed estimated six-
month mortalities of individuals within each
of the 10 imputed data sets and obtained mor-
tality estimates by averaging across them. We
assessed calibration by dividing patients into Fig. 2. Life tables of the MHOS study sample popu-
seven overlapping groups according to their lation (1998e2001 two year cohorts) vs. 2000 U.S.
predicted six-month mortality risk ($0.1, general population.
532 Han et al. Vol. 43 No. 3 March 2012

Table 1 Table 1
Study Sample Characteristics, MHOS Continued
Categorical Variables n % Categorical Variables n %

Total cases 21,870a COPD

Yes 5795 26.5
Sex
No 15,142 69.2
Male 8826 40.4
Missing 933 4.3
Female 12,921 59.1
Missing 123 0.6 Diabetes
Yes 6142 28.1
Race/ethnicity
No 15,071 68.9
Hispanic 1351 6.2
Missing 657 3.0
Non-Hispanic American Indian 186 0.9
or Alaskan Native Smoking status
Non-Hispanic Asian or Pacific 267 1.2 Never smoked 9751 44.6
Islander Former smoker 7952 36.4
Non-Hispanic Black or African 1719 7.9 Current smoker 2567 11.7
American Missing 1600 7.3
Non-Hispanic White 16,729 76.5
Any cancer
Non-Hispanic another race or 375 1.7
Yes 5796 26.5
multiracial
No 15,494 70.9
Missing 1243 5.7
Missing 580 2.7
Marital status
Married 10,447 47.8 Entire Sample
Divorced/separated/widowed 10,454 47.8
Never married 525 2.4 Continuous Variables Mean SD Missing (%)
Missing 444 2.0
Age 78.24 7.51 0 (0)
Education
b
Eighth grade or less 5264 24.1 ADLs 0.87 1.67 1215 (5.6)
High school graduate or GED/ 10,915 49.9
HRQOLc
some high school
Physical functioning 23.52 11.00 70 (0.3)
Four-year college graduate/ 4119 18.8
Roledphysical 20.13 7.73 753 (3.4)
some college or two-year
Bodily pain 32.18 10.62 439 (2.0)
degree
General health perceptions 28.06 7.91 10 (0.1)
More than a four-year college 839 3.8
Emotional well-being 37.62 13.16 426 (1.9)
degree
Social functioning 26.12 11.83 317 (1.4)
Missing 733 3.4
Energy/fatigue 32.35 9.68 381 (1.7)
Proxy status Roledemotional 23.57 19.77 1221 (5.6)
Other person to whom the 8311 38.0
MHOS ¼ Medicare Health Outcomes Survey; GED ¼ General Edu-
survey was addressed
cational Development; COPD ¼ chronic obstructive pulmonary
Person to whom the survey was 11,247 51.4 disease; ADLs ¼ activities of daily living; HRQOL ¼ health-related
addressed quality of life.
Missing 2312 10.6 a
Total n ¼ 21,870.
b
ADL measured using six items, score range 0e6; higher scores
Hypertension indicate greater functional impairment.
Yes 13,800 63.1 c
HRQOL measured using SF-36; scale scores represent standard-
No 7535 34.5 ized T-scores (mean ¼ 50, SD ¼ 10); higher scores represent
Missing 535 2.5 greater HRQOL.
Angina/coronary artery disease
Yes 7991 36.5
No 12,638 57.8 variables included ADLs, general health per-
Missing 1241 5.7
ceptions, social functioning, and energy/fa-
Congestive heart failure tigue. The c-statistic obtained from 10-fold
Yes 5341 24.4
No 15,484 70.8 cross-validation was 0.752, indicating good over-
Missing 1045 4.8 all discrimination. The model was well cali-
Other heart condition brated at lower estimated risk values; however,
Yes 7952 36.4 for estimated risk greater than 50%, it overesti-
No 12,882 58.9
Missing 1036 4.7
mated mortality, likely reflecting the small
number of events at higher risk strata (Fig. 3).
Stroke
Yes 5433 24.8 Table 3 shows the performance characteris-
No 15,433 70.6 tics of the PROMPT compared with the NHO
Missing 1004 4.6 guidelines and the SUPPORT prognostic
(Continued) model. At estimated six-month mortality risk
thresholds of 10%e70%, model sensitivity and
Vol. 43 No. 3 March 2012 Prognostic Model for Six-Month Mortality 533

Table 2
Final Multivariable Prognostic Model for
Six-Month Mortality (PROMPT)
Total Sample

Risk Factor ORa 95% CI

b c
Age 1.21 d d
Age 1 1.03 1.00 1.05
Age 2 0.93 0.85 1.03
Age 3 1.29 0.99 1.68
Sex
Male 1.54 1.41 1.67
Female (ref.) d d
Any cancer
Yes 2.96 2.72 3.21 Fig. 3. Calibration curve for actual vs. predicted six-
No (ref.) d d month mortality, PROMPT.
Congestive heart failure
Yes 1.23 1.13 1.35
No (ref.) d d
COPD Fig. 4 shows Kaplan-Meier survival curves for
Yes 1.15 1.04 1.26 respondents in different estimated risk strata.
No (ref.) d d Observed six-month mortality corresponded
Smoking status well to estimated risk, supporting the model’s
Former smoker 1.38 1.26 1.52 calibration, and by 12 months more than
Current smoker 1.31 1.13 1.52
Never smoked (ref.) d d 50% of all patients in estimated risk strata of
Proxy status 30% or more had died.
Proxy respondent 1.71 1.56 1.89
Missing 1.01 0.86 1.17
Person to whom the survey was (ref.) d d
addressed Discussion
ADLsd 1.21 1.18 1.24 In this study, we developed a new prognostic
HRQOLe model, the PROMPT, to predict six-month
General health perceptions 0.98 0.97 0.98 mortality in community-dwelling elderly pa-
Social functioning 0.99 0.98 0.99
Energy/fatigue 0.99 0.98 0.99 tients with self-reported declining health.
OR ¼ odds ratio; 95% CI ¼ 95% confidence interval; COPD ¼ chronic
The model was developed using a large diverse
obstructive pulmonary disease; ADLs ¼ activities of daily living; sample and used 11 total variables, including
HRQOL ¼ health-related quality of life.
a
Final parameter estimates (ORs) calculated using entire data sam-
HRQOL, ascertained by patient self-report.
ple (n ¼ 21,870). The model demonstrated good calibration
b
Age modeled using restricted cubic spline function with four
knots at the 5%, 35%, 65%, and 95% age percentiles given by
and discrimination overall. Importantly, diag-
CðuÞ ¼ b1 u þ q1 C1 ðuÞ þ q2 C2 ðuÞ, where C1(u) and C2(u) are cubic nostic performance at various thresholds of
terms.
c
OR is for age 75 relative to age 65, for illustrative purposes.
estimated risk was superior to existing non-
d
Odds per single unit increase in impaired ADL, where higher disease-specific models. Specificity was high
scores indicate greater functional impairment (ADL score is the
total number of activities scored as ‘‘unable to do,’’ range 0e6;
and at strata of estimated six-month mortality
zero indicates able to perform all activities, six indicates unable risk of 40% or more, the model yielded LRsþ
to perform all activities).
e
Odds per single unit increase in standardized SF-36 scale T-score
of moderate to large magnitude (5.0 or
(mean ¼ 50, SD ¼ 10); higher scores represent greater HRQOL. more) with respect to clinical predictiond
exceeding the performance of previous
models and increasing the post-test odds of
specificity were 0.8%e83.4% and 51.1% death to an extent generally considered useful
e99.99%, respectively, and corresponding in clinical decision making. The model’s PPV
PPVs and NPVs were 23.1%e64.1% and 85.3% in our study population also was high and com-
e94.5%. LRsþ exceeded 5.0 at all risk thresh- mensurate to estimated risk; 53% of patients in
olds of 40% or more, whereas LRs were near the 50% risk stratum died by six months, and
1.0. At comparable estimated risk thresholds, the proportions of observed deaths were corre-
diagnostic performance was superior to the spondingly greater in higher risk strata. On ex-
NHO and SUPPORT models. tended observation, over half of all patients
534 Han et al. Vol. 43 No. 3 March 2012

Table 3
Performance Characteristics of the PROMPT for Six-Month Mortality Compared With NHO Guidelines and
the SUPPORT Prognostic Model
Estimated Six-Month Mortality Riska

SUPPORT
PROMPT Model NHO Guidelinesb Modelc
Performance
Characteristic $0.1 $0.2 $0.3 $0.4 $0.5 $0.6 $0.7 Brd Int Nar $0.5 $0.9

Sensitivity (%) 83.4 55.1 32.9 16.8 8.0 3.7 0.8 41.7 16.2 1.4 22.1 2.4
Specificity (%) 51.1 80.2 91.7 96.7 98.8 99.5 99.9 66.7 90.1 99.5 91.4 99.4
PPV (%) 23.2 33.0 41.4 47.1 53.4 58.2 64.1 30 35 47 46 59
NPV (%) 94.5 91.0 88.5 86.8 85.8 85.3 85.0 77 76 75 78 75
LRþ 1.7 2.8 4.0 5.0 6.5 7.8 10.1 1.25 1.63 2.68 2.57 4.33
LR 0.3 0.6 0.7 0.9 0.9 1.0 1.0 0.87 0.93 0.99 0.87 0.98
NHO is now the National Hospice and Palliative Care Organization.
a
Final prognostic model based on parameter estimates calculated using entire data sample (n ¼ 21,870); pretest likelihood of six-month mortality:
MHOS sample population ¼ 15%, NHO and SUPPORT evaluation population ¼ 25%.
b
Brd, Int, and Nar: broad, intermediate, and narrow inclusion criteria for selecting patients for hospice care eligibility, based on NHO guidelines
and as operationalized by Fox et al.19 Broad criteria required $1, intermediate required $3, and narrow required $5 of a possible seven clinical
criteria specified in NHO guidelines and correspond to low, medium, and high thresholds for hospice eligibility decisions.
c
Risk categories for the SUPPORT model,19 originally expressed in terms of probability of survival (#50%, #10%), now expressed in terms of
probability of mortality.

with estimated six-month mortality risk of 30% typically considered by clinicians. Yet, in spite
or more died by 12 months. of these significant constraints on prognostic
These promising performance characteris- power, the PROMPT still demonstrated supe-
tics are particularly noteworthy given the types rior performance compared with existing tools,
of variables used in our model and the nature such as the SUPPORT model and NHO guide-
of our study population. Unlike prior prognos- lines. This supports the model’s robustness
tic efforts, the PROMPT included no physio- and potential transportability to more narrowly
logic or laboratory data and relatively little defined populations with higher pretest proba-
clinical data regarding disease characteristics bilities of mortality in which prognostic perfor-
or health services utilization. The study popula- mance would likely be more optimal. These
tion was clinically heterogeneous, ambulatory, conclusions remain preliminary, however, be-
community dwelling, and relatively healthy, cause our model has yet to be externally vali-
with a lower pretest mortality risk, compared dated and directly compared with other tools.
with populations included in other prognostic The primary limitation of the PROMPT is
modeling efforts and for whom hospice care is one shared by all existing prognostic tools for
predicting short-term mortality: insufficient
sensitivity to ‘‘rule out’’ death in a substantial
proportion of patients. This is not surprising
given that there are undoubtedly numerous
causal factors and trajectories in the dying
process,72,73 and no prognostic model has ac-
counted for them all. For the PROMPT, fur-
thermore, comorbidities were ascertained by
self-report only and some important ones, for
example, dementia and renal and liver disease,
were not assessed. A substantial proportion of
surveys (38%) also were completed by proxy,
presumably because patients were too ill or im-
paired to do so themselves. Finally, our selec-
tion of patients on the basis of self-reported
Fig. 4. Kaplan-Meier survival curves for MHOS re- decline in health likely also led to the inclu-
spondents in different model-estimated six-month sion of patients with acute, self-limited condi-
mortality risk strata. tions with little impact on mortality. All these
Vol. 43 No. 3 March 2012 Prognostic Model for Six-Month Mortality 535

factors likely limit the PROMPT’s sensitivity physicians’ clinical reluctance to render prog-
and use as an exclusive means of determining noses,22,75 and patients’ reluctance to accept
hospice eligibility because this would result in them.76 The PROMPT’s ability to identify immi-
the denial of hospice services for most dying nently dying patients with very few false posi-
patients. This limitation has led other mod- tives addresses this concern, providing
elers to conclude that the goal of determining physicians and patients with the necessary reas-
individuals’ risk of six-month mortality is surance to make critical decisions about end-of-
unrealistic.19,25 life care.
Yet, we believe that our modeling effort of- A final limitation of the PROMPT’s perfor-
fers important insights for future research mance is its weaker calibration at higher esti-
and that the PROMPT has significant potential mated risk levels, at which it overestimated
utility for clinical care. Our study adds to mortality. This is likely a consequence of the
mounting evidence of the prognostic power small number of total deaths in these sub-
of HRQOL. The PROMPT’s superior overall groups, reflecting the low overall mortality
performance compared with efforts incorpo- rate of the study sample (15%). In sicker pop-
rating disease and physiologic variables alone ulations with a higher pretest likelihood of
supports the hypothesis that as death ap- mortality, it is possible that the model’s calibra-
proaches, HRQOL assumes greater prognostic tion would be improved.
significance.32,33 The prominent role of simi- However, this remains to be seen, and fur-
lar HRQOL variables in other prognostic ther evaluation is needed before the PROMPT
models in elderly patients with advanced ill- can be implemented clinically. Although the
ness24,74 further bears this out, supporting large size and geographic and clinical hetero-
the PROMPT’s validity and the value of inte- geneity of the study population enhances the
grating HRQOL in future modeling efforts. model’s generalizability, it needs to be vali-
Furthermore, despite its low sensitivity in rul- dated prospectively in other populations with
ing out imminent death, the PROMPT has sig- differing comorbidities and experiences with
nificant potential to improve end-of-life care health care. The target population of any
given the prevailing underutilization of hospice predictive model determines both its clinical
services, overutilization of life-prolonging inter- appropriateness and performance characteris-
ventions, and lack of more accurate, evidence- tics, including sensitivity and specificity,77 and
based, and explicit prognostic methods. These ours consisted of community-dwelling elders
circumstances alone raise the possibility that with self-reported declining health. However,
use of the model could increase hospice utiliza- the PROMPT might be more accurate and use-
tion and advance care planning. Yet, the ful in alternative populations, for example,
PROMPT’s greatest potential value lies in its patients identified on the basis of comorbid-
ability to confirm a poor six-month prognosis. ities and health care utilization (as in the
Its very high specificity across a range of esti- SUPPORT study19,21 and more recent prognos-
mated mortality risks (97% or higher for all es- tic model efforts in nursing home patients with
timated risk cut points of 40% or greater) dementia24) or physicians’ own prognostic es-
makes the PROMPT an extremely valuable timates,78,79 recently operationalized through
tool for ‘‘ruling in’’ imminent death, with very what has been termed the ‘‘surprise ques-
few false positives. From both an ethical and tion:’’80e82 ‘‘Would I be surprised if this patient
a clinical standpoint, this function has at least died in the next 12 months?’’ Applying our
as much clinical importance as ruling out tool in such selected populationsdwith higher
death. The potential harm of a false negative es- pretest probabilities of six-month mortalityd
timate of six-month mortality is overly aggres- would likely improve prognostic performance.
sive care at the end of life. Although Future research also might fruitfully examine
undesirable, this outcome is arguably more tol- whether testing strategies combining multiple
erable than the potential irreversible harm of prognostic tools and factors could further en-
a false positive estimate: mistakenly labeling pa- hance prognostic power. For example, sensitiv-
tients as ‘‘dying’’ and forgoing potentially bene- ity might be increased by using the PROMPT
ficial or curative interventions. This ethical in parallel with other approaches, such as
concern may be an important reason for physicians’ prognostic estimates or the CMS
536 Han et al. Vol. 43 No. 3 March 2012

Local Coverage Determination guidelines83 IOM/itemdetail.asp?itemID¼CMS012673. Accessed

used by hospice providers to determine hos- October 27, 2011.
pice eligibility. 2. Brody H, Lynn J. The physician’s responsibility
Finally, more work is needed not only to under the new Medicare reimbursement for hospice
evaluate and improve the PROMPT but to un- care. N Engl J Med 1984;310:920e922.
derstand the optimal means and outcomes of 3. Christakis NA, Escarce JJ. Survival of Medicare
applying prognostic models clinically. The patients after enrollment in hospice programs.
landmark SUPPORT study demonstrated that N Engl J Med 1996;335:172e178.
simply providing physicians with prognostic in- 4. Emanuel EJ, Ash A, Yu W, et al. Managed care,
formation may not alter end-of-life decision hospice use, site of death, and medical expenditures
in the last year of life. Arch Intern Med 2002;162:
making or patterns of care.84 Various factors, 1722e1728.
including physician and patient attitudes14,22
and the structures and processes of health 5. McCarthy EP, Burns RB, Ngo-Metzger Q,
Davis RB, Phillips RS. Hospice use among Medicare
care may limit effective utilization of prognos- managed care and fee-for-service patients dying with
tic models. The feasibility of implementing cancer. JAMA 2003;289:2238e2245.
a PRO-based model such as ours, which uses 6. Connor SR, Elwert F, Spence C, Christakis NA.
11 variables but 28 individual data elements, Geographic variation in hospice use in the United
remains to be determined. These and other States in 2002. J Pain Symptom Manage 2007;34:
potential barriers need to be better under- 277e285.
stood, along with the appropriate methods 7. Flory J, Yinong YX, Gurol I, et al. Place of death:
for communicating prognostic information in U.S. trends since 1980. Health Aff (Millwood) 2004;
a sensitive, comprehensible manner. The cur- 23:194e200.
rent effort provides a foundation for such 8. Angus DC, Barnato AE, Linde-Zwirble WT, et al.
work and efforts to refine existing models Use of intensive care at the end of life in the United
and determine optimal strategies for their States: an epidemiologic study. Crit Care Med 2004;
32:638e643.
implementation.
9. Barnato AE, McClellan MB, Kagay CR,
Garber AM. Trends in inpatient treatment intensity
among Medicare beneficiaries at the end of life.
Disclosures and Acknowledgments Health Serv Res 2004;39:363e375.
This study was supported by intramural 10. Steinhauser KE, Christakis NA, Clipp EC, et al.
research funds from the National Cancer Insti- Factors considered important at the end of life by
patients, family, physicians, and other care pro-
tute, National Institutes of Health. Ron Hays viders. JAMA 2000;284:2476e2482.
was supported in part by National Institute
on Aging grants (P30AG021684 and P30- 11. Steinhauser KE, Clipp EC, McNeilly M, et al. In
search of a good death: observations of patients,
AG028748) and a National Center on Mino- families, and providers. Ann Intern Med 2000;132:
rity Health and Health Disparities grant 825e832.
(2P20MD000182). The authors declare no 12. Higginson IJ, Sen-Gupta GJ. Place of care in ad-
conflicts of interest. vanced cancer: a qualitative systematic literature re-
The findings and conclusions in this report view of patient preferences. J Palliat Med 2000;3:
are those of the authors and do not necessarily 287e300.
represent the views of the National Cancer 13. Friedman BT, Harwood MK, Shields M. Barriers
Institute. and enablers to hospice referrals: an expert over-
The authors thank Robert Arnold, Rachel view. J Palliat Med 2002;5:73e84.
Ballard-Barbash, Steve Clauser, and anonymous 14. Lamont EB. A demographic and prognostic ap-
reviewers for helpful comments on an earlier proach to defining the end of life. J Palliat Med
version of the manuscript. 2005;8(Suppl 1):S12eS21.
15. Brickner L, Scannell K, Marquet S, Ackerson L.
Barriers to hospice care and referrals: survey of phy-
sicians’ knowledge, attitudes, and perceptions in
References a health maintenance organization. J Palliat Med
2004;7:411e418.
1. Centers for Medicare and Medicaid Services
(CMS). Pub. 100-02 Medicare benefit policy manual. 16. Christakis NA, Lamont EB. Extent and determi-
2004. Available from http://www.cms.gov/Manuals/ nants of error in physicians’ prognoses in terminally
Vol. 43 No. 3 March 2012 Prognostic Model for Six-Month Mortality 537

ill patients: prospective cohort study. West J Med Longitudinal Project. J Am Geriatr Soc 2000;48(5
2000;172:310e313. Suppl):S16eS24.
17. Chow E, Harth T, Hruby G, et al. How accurate 30. Walter LC, Brand RJ, Counsell SR, et al. Devel-
are physicians’ clinical predictions of survival and opment and validation of a prognostic index for
the available prognostic tools in estimating survival 1-year mortality in older adults after hospitalization.
times in terminally ill cancer patients? A systematic JAMA 2001;285:2987e2994.
review. Clin Oncol (R Coll Radiol) 2001;13:209e218.
31. Lee SJ, Lindquist K, Segal MR, Covinsky KE. De-
18. Stuart B. The NHO medical guidelines for non- velopment and validation of a prognostic index for
cancer disease and local medical review policy: hos- 4-year mortality in older adults. JAMA 2006;295:
pice access for patients with diseases other than can- 801e808.
cer. Hosp J 1999;14:139e154.
32. Mackillop WJ. Differences in prognostication
19. Fox E, Landrum-McNiff K, Zhong Z, et al. Eval- between early and advanced cancer. In: Glare P,
uation of prognostic criteria for determining hos- Christakis NA, eds. Prognosis in advanced cancer.
pice eligibility in patients with advanced lung, New York: Oxford University Press, 2008:13e23.
heart, or liver disease. SUPPORT Investigators.
33. Glare P, Christakis NA. Overview: advancing the
Study to Understand Prognoses and Preferences
clinical science of prognostication. In: Glare P,
for Outcomes and Risks of Treatments. JAMA
Christakis NA, eds. Prognosis in advanced cancer.
1999;282:1638e1645.
New York: Oxford University Press, 2008:3e12.
20. Schonwetter RS, Han B, Small BJ, et al. Predictors
34. Dorr DA, Jones SS, Burns L, et al. Use of health-
of six-month survival among patients with dementia:
related, quality-of-life metrics to predict mortality
an evaluation of hospice Medicare guidelines. Am J
and hospitalizations in community-dwelling seniors.
Hosp Palliat Care 2003;20:105e113.
J Am Geriatr Soc 2006;54:667e673.
21. Knaus WA, Harrell FE Jr, Lynn J, et al. The
35. Coates A, Porzsolt F, Osoba D. Quality of life in
SUPPORT prognostic model. Objective estimates
oncology practice: prognostic value of EORTC
of survival for seriously ill hospitalized adults. Study
QLQ-C30 scores in patients with advanced malig-
to understand prognoses and preferences for out-
nancy. Eur J Cancer 1997;33:1025e1030.
comes and risks of treatments. Ann Intern Med
1995;122:191e203. 36. Vigano A, Donaldson N, Higginson IJ, et al.
Quality of life and survival prediction in terminal
22. Glare PA, Sinclair CT. Palliative medicine re-
cancer patients: a multicenter study. Cancer 2004;
view: prognostication. J Palliat Med 2008;11:
101:1090e1098.
84e103.
37. Rumsfeld JS, MaWhinney S, McCarthy M Jr,
23. Chow E, Fung K, Panzarella T, et al. A predictive
et al. Health-related quality of life as a predictor of
model for survival in metastatic cancer patients at-
mortality following coronary artery bypass graft sur-
tending an outpatient palliative radiotherapy clinic.
gery. Participants of the Department of Veterans
Int J Radiat Oncol Biol Phys 2002;53:1291e1302.
Affairs Cooperative Study Group on Processes,
24. Mitchell SL, Kiely DK, Hamel MB, et al. Estimat- Structures, and Outcomes of Care in Cardiac Sur-
ing prognosis for nursing home residents with ad- gery. JAMA 1999;281:1298e1303.
vanced dementia. JAMA 2004;291:2734e2740.
38. Chang VT, Thaler HT, Polyak TA, et al. Quality
25. Mitchell SL, Miller SC, Teno JM, Davis RB, of life and survival: the role of multidimensional
Shaffer ML. The advanced dementia prognostic symptom assessment. Cancer 1998;83:173e179.
tool: a risk score to estimate survival in nursing
39. Carey EC, Walter LC, Lindquist K, Covinsky KE.
home residents with advanced dementia. J Pain
Development and validation of a functional morbid-
Symptom Manage 2010;40:639e651.
ity index to predict mortality in community-dwelling
26. Mitchell SL, Miller SC, Teno JM, et al. Predic- elders. J Gen Intern Med 2004;19:1027e1033.
tion of 6-month survival of nursing home residents
40. Meyer F, Fortin A, Gelinas M, et al. Health-
with advanced dementia using ADEPT vs. hospice
related quality of life as a survival predictor for pa-
eligibility guidelines. JAMA 2010;304:1929e1935.
tients with localized head and neck cancer treated
27. Anderson F, Downing GM, Hill J, Casorso L, with radiation therapy. J Clin Oncol 2009;27:
Lerch N. Palliative performance scale (PPS): a new 2970e2976.
tool. J Palliat Care 1996;12:5e11.
41. Dancey J, Zee B, Osoba D, et al. Quality of life
28. Morita T, Tsunoda J, Inoue S, Chihara S. The scores: an independent prognostic variable in a gen-
Palliative Prognostic Index: a scoring system for sur- eral population of cancer patients receiving chemo-
vival prediction of terminally ill cancer patients. therapy. The National Cancer Institute of Canada
Support Care Cancer 1999;7:128e133. Clinical Trials Group. Qual Life Res 1997;6:
151e158.
29. Teno JM, Harrell FE Jr, Knaus W, et al. Predic-
tion of survival for older hospitalized patients: the 42. Inouye SK, Peduzzi PN, Robison JT, et al. Im-
HELP survival model. Hospitalized Elderly portance of functional measures in predicting
538 Han et al. Vol. 43 No. 3 March 2012

mortality among older hospitalized patients. JAMA The role of length of stay and comorbidities.
1998;279:1187e1193. JAMA 1988;260:2240e2246.
43. Cunningham WE, Crystal S, Bozzette S, 58. Parkerson GR Jr, Gutman RA. Health-related
Hays RD. The association of health-related quality quality of life predictors of survival and hospital uti-
of life with survival among persons with HIV infec- lization. Health Care Financ Rev 2000;21:171e184.
tion in the United States. J Gen Intern Med 2005;
20:21e27. 59. Kimmel PL, Peterson RA, Weihs KL, et al. Psy-
chosocial factors, behavioral compliance and sur-
44. Selim AJ, Berlowitz DR, Fincke G, et al. Risk- vival in urban hemodialysis patients. Kidney Int
adjusted mortality rates as a potential outcome indi- 1998;54:245e254.
cator for outpatient quality assessments. Med Care
2002;40:237e245. 60. Riis J, Loewenstein G, Baron J, et al. Ignorance
of hedonic adaptation to hemodialysis: a study using
45. Covinsky KE, Justice AC, Rosenthal GE, ecological momentary assessment. J Exp Psychol
Palmer RM, Landefeld CS. Measuring prognosis Gen 2005;134:3e9.
and case mix in hospitalized elders. The importance
of functional status. J Gen Intern Med 1997;12: 61. Sprangers MA, Schwartz CE. Reflections on
203e208. changeability versus stability of health-related qual-
ity of life: distinguishing between its environmental
46. Vigano A, Dorgan M, Buckingham J, Bruera E, and genetic components. Health Qual Life Out-
Suarez-Almazor ME. Survival prediction in terminal comes 2008;6:89.
cancer patients: a systematic review of the medical
literature. Palliat Med 2000;14:363e374. 62. Reilly BM, Evans AT. Translating clinical re-
search into clinical practice: impact of using predic-
47. Schonberg MA, Davis RB, McCarthy EP, tion rules to make decisions. Ann Intern Med 2006;
Marcantonio ER. Index to predict 5-year mortality 144:201e209.
of community-dwelling adults aged 65 and older us-
ing data from the national health interview survey. 63. Laupacis A, Sekar N, Stiell IG. Clinical predic-
J Gen Intern Med 2009;24:1115e1122. tion rules. A review and suggested modifications of
methodological standards. JAMA 1997;277:
48. Cooper JK, Kohlmann T, Michael JA, Haffer SC, 488e494.
Stevic M. Health outcomes. New quality measure for
Medicare. Int J Qual Health Care 2001;13:9e16. 64. Hadorn DC, Keeler EB, Rogers WH, Brook RH.
Assessing the performance of mortality prediction
49. Jones N 3rd, Jones SL, Miller NA. The Medicare
models. Santa Monica, CA: RAND, 1993.
Health Outcomes Survey program: overview, con-
text, and near-term prospects. Health Qual Life 65. Schonberg MA, Davis RB, McCarthy EP,
Outcomes 2004;2:33. Marcantonio ER. Index to predict 5-year mortality
of community-dwelling adults aged 65 and older us-
50. Haffer SC, Bowen SE. Measuring and improving
ing data from the National Health Interview Survey.
health outcomes in Medicare: the Medicare HOS
J Gen Intern Med 2009;24:1115e1122.
program. Health Care Financ Rev 2004;25:1e3.
51. Ware JE Jr, Sherbourne CD. The MOS 36-item 66. Rubin DB. Multiple imputation for non-
short-form health survey (SF-36). I. Conceptual response in surveys. New York: Wiley, 1987.
framework and item selection. Med Care 1992;30: 67. Ng JH, Kasper JD, Forrest CB, Bierman AS. Pre-
473e483. dictors of voluntary disenrollment from Medicare
52. Heron M, Hoyert D, Murphy S, et al. Deaths: fi- managed care. Med Care 2007;45:513e520.
nal data for 2006. Natl Vital Stat Rep 2009;57:1e134. 68. Cutler SJ, Ederer F. Maximum utilization of the
53. Atkinson AC. A note on the generalized infor- life table method in analyzing survival. J Chronic Dis
mation criterion for choice of a model. Biometrika 1958;8:699e712.
1980;67:413e418. 69. Dignam JJ, Huang L, Ries L, et al. Estimating
54. Sauerbrei W. The use of resampling methods to breast cancer-specific and other-cause mortality in
simplify regression models in medical statistics. clinical trial and population-based cancer registry
Appl Stat 1999;48:313e329. cohorts. Cancer 2009;115:5272e5283.
55. Vergouwe Y, Royston P, Moons KG, Altman DG. 70. Subramanian J, Simon R. An evaluation of re-
Development and validation of a prediction model sampling methods for assessment of survival risk
with missing predictor data: a practical approach. prediction in high-dimensional settings. Stat Med
J Clin Epidemiol 2010;63:205e214. 2011;30:642e653.
56. Iezzoni LI, Foley SM, Daley J, et al. Comorbidities, 71. Harrell FE, Lee KL, Mark DB. Multivariable
complications, and coding bias. Does the number prognostic models: issues in developing models,
of diagnosis codes matter in predicting in-hospital evaluating assumptions and adequacy, and measur-
mortality? JAMA 1992;267:2197e2203. ing and reducing errors. Stat Med 1996;15:361e387.
57. Jencks SF, Williams DK, Kay TL. Assessing 72. Dy S, Lynn J. Getting services right for those
hospital-associated deaths from discharge data. sick enough to die. BMJ 2007;334:511e513.
Vol. 43 No. 3 March 2012 Prognostic Model for Six-Month Mortality 539

73. Murray SA, Kendall M, Boyd K, Sheikh A. Illness 80. Moss AH, Ganjoo J, Sharma S, et al. Utility of the
trajectories and palliative care. BMJ 2005;330: ‘‘surprise’’ question to identify dialysis patients with high
1007e1011. mortality. Clin J Am Soc Nephrol 2008;3:1379e1384.
74. Flacker JM, Kiely DK. Mortality-related factors 81. Cohen LM, Ruthazer R, Moss AH, Germain MJ.
and 1-year survival in nursing home residents. Predicting six-month mortality for patients who are
J Am Geriatr Soc 2003;51:213e221. on maintenance hemodialysis. Clin J Am Soc Neph-
75. Christakis NA. Death foretold: Prophecy and rol 2010;5:72e79.
prognosis in medical care. Chicago, IL: University 82. Pattison M, Romer AL. Improving care
of Chicago Press, 1999. through the end of life: launching a primary care
76. Fried TR, Drickamer M. Garnering support for clinic-based program. J Palliat Med 2001;4:
advance care planning. JAMA 2010;303:269e270. 249e254.
77. Moons KG, Harrell FE. Sensitivity and specificity 83. Centers for Medicare and Medicaid Services.
should be de-emphasized in diagnostic accuracy LCD for hospice determining terminal status
studies. Acad Radiol 2003;10:670e672. (L13653). 2009. Available from http://www.cms.hhs.
78. Glare P, Virik K, Jones M, et al. A systematic re- gov/mcd/viewlcd.asp?lcd_id¼13653&lcd_version¼
view of physicians’ survival predictions in terminally 30&show¼all. Accessed March 30, 2010.
ill cancer patients. BMJ 2003;327:195e198. 84. The SUPPORT Principal Investigators.
79. Maltoni M, Caraceni A, Brunelli C, et al. Prognos- A controlled trial to improve care for seriously ill
tic factors in advanced cancer patients: evidence- hospitalized patients. The Study to Understand
based clinical recommendationsea study by the Prognoses and Preferences for Outcomes and Risks
Steering Committee of the European Association of Treatments (SUPPORT). JAMA 1995;274:
for Palliative Care. J Clin Oncol 2005;23:6240e6248. 1591e1598.