Advertisement

Brief English and Spanish Survey Detects Change in Response to Advance Care Planning Interventions

Open ArchivePublished:September 17, 2019DOI:https://doi.org/10.1016/j.jpainsymman.2019.09.004

      Abstract

      Context

      The validated 82-item Advance Care Planning (ACP) Engagement Survey measures a broad range of ACP behaviors but is long.

      Objectives

      Determine whether shorter survey versions (55-item, 34-item, 15-item, 9-item, and 4-item versions) can detect similar change in response to two well-validated ACP interventions and provide practical effect size information.

      Methods

      We assessed ACP engagement for 986 English- and Spanish-speaking adults in a randomized trial of PREPARE vs. an advance directive-only study arms. The survey was administered at baseline, one week, three months, six months, and 12 months. We calculated mean change scores from baseline to follow-up time points by study arm, intraclass correlation coefficients of change scores between the 82-item survey with shorter versions, and within-group and between-group effect sizes of the mean change scores.

      Results

      Shorter survey versions were able to detect within-group and between-group changes at all time points. Within-group intraclass correlations of the 82-item to shorter versions were high (0.78–0.97), and the amount of between-group differences was comparable using all survey versions. Twelve-month within-group effect sizes ranged narrowly from 0.76 to 1.05 for different survey versions in the PREPARE arm and from 0.44 to 0.64 for the advance directive-only version. Between-group effect sizes ranged narrowly from 0.24 to 0.30 for different survey versions. Results were similar when stratified by English and Spanish speakers.

      Conclusion

      Shorter versions of the ACP Engagement Survey were able to detect within-group and between-group changes comparable with the 82-item version and can be useful for efficiently and effectively measuring ACP engagement in research and clinical settings.

      Key Words

      Introduction

      Advance care planning (ACP) has garnered increasing attention from health systems and researchers because it has been shown to improve patients' satisfaction with medical care and increase agreement of patients' wishes for care received.
      • Klingler C.
      • In der Schmitten J.
      • Marckmann G.
      Does facilitated advance care planning reduce the costs of care near the end of life? Systematic review and ethical considerations.
      • Silveira M.J.
      • Kim S.Y.
      • Langa K.M.
      Advance directives and outcomes of surrogate decision making before death.
      • Hammes B.J.
      • Rooney B.L.
      • Gundrum J.D.
      A comparative, retrospective, observational study of the prevalence, availability, and specificity of advance care plans in a county that implemented an advance care planning microsystem.
      • Detering K.M.
      • Hancock A.D.
      • Reade M.C.
      • Silvester W.
      The impact of advance care planning on end of life care in elderly patients: randomised controlled trial.
      • Pope T.M.
      Legal briefing: medicare coverage of advance care planning.
      Historically, most studies have focused solely on the completion of an advance directive (AD) to measure successful ACP. However, several studies have shown that ACP is a complex process that occurs over time and involves multiple discrete behaviors.
      • Sudore R.L.
      • Fried T.R.
      Redefining the ‘‘planning” in advance care planning: preparing for end-of-life decision making.
      • Sudore R.L.
      • Schickedanz A.D.
      • Landefeld C.S.
      • et al.
      Engagement in multiple steps of the advance care planning process: a descriptive study of diverse older adults.
      • Fried T.R.
      • Redding C.A.
      • Robbins M.L.
      • et al.
      Stages of change for the component behaviors of advance care planning.
      • Fried T.R.
      • Redding C.A.
      • Robbins M.L.
      • et al.
      Promoting advance care planning as health behavior change: development of scales to assess decisional balance, medical and religious beliefs, and processes of change.
      • Sudore R.L.
      • Heyland D.K.
      • Lum H.D.
      • et al.
      Outcomes that define successful advance care planning: a Delphi panel consensus.
      Studies have also shown that people are in varying stages of readiness to engage in these behaviors.
      • Sudore R.L.
      • Knight S.J.
      • McMahan R.D.
      • et al.
      A novel website to prepare diverse older adults for decision making and advance care planning: a pilot study.
      ,
      • Lum H.D.
      • Barnes D.E.
      • Katen M.T.
      • et al.
      Improving a full range of advance care planning behavior change and action domains: the PREPARE randomized trial.
      The ACP Engagement Survey was developed, culturally vetted, and validated to measure the complex process of ACP.
      • Sudore R.L.
      • Stewart A.L.
      • Knight S.J.
      • et al.
      Development and validation of a questionnaire to detect behavior change in multiple advance care planning behaviors.
      ,
      • Sudore R.L.
      • Heyland D.K.
      • Barnes D.E.
      • et al.
      Measuring advance care planning: optimizing the advance care planning engagement survey.
      The survey is based on Social Cognitive and Behavior Change theories and focuses on four behavior change constructs (i.e., knowledge, contemplation, self-efficacy, and readiness) within four ACP domains (i.e., surrogate decision makers, values and quality of life, flexibility in surrogate decision making, and asking doctors questions). Although validated and shown to detect change in response to ACP interventions, the 82-item version of the survey takes 50 minutes to administer,
      • Sudore R.L.
      • Heyland D.K.
      • Barnes D.E.
      • et al.
      Measuring advance care planning: optimizing the advance care planning engagement survey.
      ,
      • Howard M.
      • Bonham A.J.
      • Heyland D.K.
      • et al.
      Measuring engagement in advance care planning: a cross-sectional multicentre feasibility study.
      reducing its utility. Brief, feasible, and validated surveys that can effectively measure the ACP process and can detect change in response to ACP interventions are needed for research and clinical programs.
      In a prior study, we conducted item reduction and validated five progressively shorter versions of the ACP Engagement Survey, including a 55-item, 34-item, 15-item, 9-item, and 4-item version.
      • Sudore R.L.
      • Heyland D.K.
      • Barnes D.E.
      • et al.
      Measuring advance care planning: optimizing the advance care planning engagement survey.
      However, that prior study used blinded trial data with a small sample size and only accessed pre-to-post changes during a one-week follow-up period.
      The present study builds on that prior work by including larger complete trial cohort data of English- and Spanish-speaking older adults from a published randomized controlled trial designed to compare two well-validated interventions.
      • Sudore R.L.
      • Schillinger D.
      • Katen M.T.
      • et al.
      Engaging diverse English- and Spanish-speaking older adults in advance care planning: the PREPARE randomized clinical trial.
      Follow-up time points now include one week, three months, six months, and 12 months, and we calculate both within-group and between-group differences by study arm. This study also provides practical effect size information for the use of brief, literacy-appropriate, English and Spanish, culturally vetted measures for a range of ACP behaviors. We will evaluate if change scores in response to an ACP intervention for progressively shorter versions of the survey, including a four-item version, are highly correlated with the original 82-item version.

      Methods

      Data Sources and Participants

      Study participants included 986 English- and Spanish-speaking patients enrolled in a randomized trial at the San Francisco General Hospital from February 2014 to November 2017. These participants were randomly assigned to two intervention groups, an easy-to-read AD written at the fifth-grade reading level (AD-only arm) and the PREPARE Web site (PREPAREforYourCare.org) plus the AD (PREPARE arm). PREPARE is an interactive online ACP program that uses video stories to help people identify their wishes for medical care and models how to discuss those wishes with others. The trial compared the efficacy of PREPARE plus the easy-to-read AD vs. the AD alone to engage participants in the ACP process. The study was approved by the University of California, San Francisco Institutional Review Board, and the trial has been published.
      • Sudore R.L.
      • Schillinger D.
      • Katen M.T.
      • et al.
      Engaging diverse English- and Spanish-speaking older adults in advance care planning: the PREPARE randomized clinical trial.
      ,
      • Sudore R.L.
      • Barnes D.E.
      • Le G.M.
      • et al.
      Improving advance care planning for English-speaking and Spanish-speaking older adults: study protocol for the PREPARE randomized controlled trial.

      Outcomes and Measures

      The validated patient-reported ACP Engagement 82-item Survey includes 57 items concerning behavior change processes (i.e., knowledge, contemplation, self-efficacy, and readiness) measured on an average five-point Likert scale and 25 ACP action items such as discussing and documenting ACP wishes using yes or no response options. The survey scores were unweighted on a one- to five-point scale with higher scores reflecting greater ACP engagement (response options: 1—not at all, 2—a little, 3—somewhat, 4—fairly, and 5—extremely for knowledge, self-efficacy, and readiness subscales, and 1—never, 2—once or twice, 3—a few times, 4—several times, and 5—a lot for the contemplation subscale). A detailed table including the questions and response options of the original version and shorter versions (i.e., 55-, 34-, 15-, 9-, and 4-item versions) has been published.
      • Sudore R.L.
      • Heyland D.K.
      • Barnes D.E.
      • et al.
      Measuring advance care planning: optimizing the advance care planning engagement survey.
      The survey was shortened based on multiple criteria.
      • Sudore R.L.
      • Heyland D.K.
      • Barnes D.E.
      • et al.
      Measuring advance care planning: optimizing the advance care planning engagement survey.
      The 25 ACP action items were removed in all shorter versions because of redundancy as yes/no actions can also be calculated from the readiness questions, which assessed readiness to discuss/document with surrogate decision makers, discuss/document wishes for medical care, discuss/document flexibility for the surrogate, and ask doctors questions. This resulted in all five shorter survey versions measured on an average five-point Likert scale for the ACP engagement score. The behavior change process questions concerning contemplation and questions concerning flexibility in surrogate decision making and asking doctors questions were the questions most often deleted from shorter versions. To be able to compare the average five-point Likert scores of the shorter versions with the original version, the overall average ACP engagement score for the 82-item survey was created by averaging the five-point Likert scales for the process measures and also for the action measures by assigning a value of five to response options of yes and a value of 0 to response options of no.
      For the randomized trial, we administered the full 82-item ACP Engagement Survey at baseline, one week, three months, six months, and 12 months. We also assessed self-reported participant characteristics at baseline, including age, gender, race/ethnicity, education, health literacy, finances, and health status.
      • Sudore R.L.
      • Schillinger D.
      • Katen M.T.
      • et al.
      Engaging diverse English- and Spanish-speaking older adults in advance care planning: the PREPARE randomized clinical trial.
      In the trial, participants were block randomized by adequate vs. limited health literacy using a random number generator.
      • Chew L.D.
      • Griffin J.M.
      • Partin M.R.
      • et al.
      Validation of screening questions for limited health literacy in a large VA outpatient population.
      Prior ACP documentation before the baseline interview was obtained using a composite of any prior legal forms and documented discussions about ACP within the past five years by chart review.
      • Sudore R.L.
      • Fried T.R.
      Redefining the ‘‘planning” in advance care planning: preparing for end-of-life decision making.
      ,
      • Howard M.
      • Bonham A.J.
      • Heyland D.K.
      • et al.
      Measuring engagement in advance care planning: a cross-sectional multicentre feasibility study.

      Statistical Analysis

      We first compared baseline characteristics of the AD-only and PREPARE arms overall and stratified by English and Spanish speakers using unpaired t-tests for continuous variables and Chi-squared tests for categorical variables. In addition, we compared baseline characteristics of English and Spanish speakers overall. We then assessed the ability of each survey version to detect change in average ACP Behavior Change Survey scores in response to ACP interventions. We used mixed-effects repeated-measures model for the average ACP engagement score with fixed effects on time (i.e., baseline, one week, etc.), intervention group (PREPARE vs. AD-only arms), and group-by-time interaction, with time as a categorical variable to allow for nonlinearity of responses over time. All analyses were adjusted for the randomization block variable of health literacy and prior ACP documentation and clustered by physician.
      • Sudore R.L.
      • Schillinger D.
      • Katen M.T.
      • et al.
      Engaging diverse English- and Spanish-speaking older adults in advance care planning: the PREPARE randomized clinical trial.
      We then calculated mean change scores from baseline to each of the four follow-up time points (i.e., one week, and three, six, and 12 months) by study arm and measured within-group effect sizes (Cohen's d) of the mean change scores. Using the original 82-item survey as the reference, we computed the intraclass correlation coefficients of the change in survey scores at each time point for progressively shorter versions. Finally, we evaluated between-group effect sizes and mean change score differences between PREPARE vs. AD-only arms for each shorter survey version at each follow-up time point and compared them with the original 82-item version using t-tests. All analyses were also stratified by English- and Spanish-speaking participants. We used SAS 9.4 statistical software (SAS Institute, Inc, Cary, NC) and STATA 15.1 (StataCorp LLC, College Station, TX); all tests of statistical significance were two sided, and we conducted Bonferroni adjustment for multiple between-group comparisons.

      Results

      Among 986 enrolled participants, 505 were randomized to the AD-only arm and 481 to the PREPARE arm. The mean age of overall participants was 63.3 (6.4) years, 603 (61.2%) were women, 634 (64.3%) were nonwhite, 445 (45.1%) were Spanish speakers, 387 (39.3%) had limited health literacy, 504 (51.1%) reported fair-to-poor health status, and 269 (27.3%) had prior ACP documentation (Table 1). Participant characteristics did not differ between study arms overall or by English or Spanish speakers. For the overall cohort including both arms, Spanish-speaking participants were more likely than English-speaking participants to be women, have less education, have higher rates of limited health literacy, and worse self-rated health (P < 0.05; Table 1).
      Table 1Baseline Participant Characteristics
      Participant CharacteristicAll Participants (N = 986)English Speakers (N = 541)Spanish Speakers (N = 445)
      n (%)
      AD-Only (n = 505)PREPARE (n = 481)AD-Only (n = 279)PREPARE (n = 262)AD-Only (n = 226)PREPARE (n = 219)
      Age, mean (SD)63.1 (6.3)63.5 (6.4)62.3 (5.4)63.0 (6.1)64.1 (7.2)64.1 (6.8)
      Gender
       Women314 (62.2)289 (60.1)151 (54.1)132 (50.4)163 (72.1)157 (71.7)
       Men191 (37.8)192 (39.9)128 (45.9)130 (49.6)63 (27.9)62 (28.3)
      Race/ethnicity
       White Latino or Hispanic248 (49.1)251 (52.2)24 (8.6)35 (13.4)224 (99.1)216 (98.6)
       White non-Latino104 (20.6)85 (17.7)104 (37.3)84 (32.1)01 (0.5)
       African American92 (18.2)86 (17.9)92 (33.0)86 (32.8)00
       Asian/Pacific Islander34 (6.7)44 (9.1)34 (12.2)44 (16.8)00
       Multiethnic/other27 (5.4)15 (3.1)25 (8.9)13 (4.9)2 (0.9)2 (0.9)
      High school education or less287 (56.8)289 (60.1)102 (36.6)102 (38.9)185 (81.9)187 (85.4)
      Limited health literacy
      Missing values: limited health literacy: 1.1% missing; finances: 1.6% missing; and self-rated health: 0.3% missing.
      202 (40.3)185 (39.0)60 (21.7)56 (21.7)142 (63.1)129 (59.7)
      Finances, not enough to make ends meet
      Missing values: limited health literacy: 1.1% missing; finances: 1.6% missing; and self-rated health: 0.3% missing.
      124 (25.0)119 (25.1)65 (23.8)56 (21.5)59 (26.5)63 (29.4)
      Self-rated health, fair-to-poor
      Missing values: limited health literacy: 1.1% missing; finances: 1.6% missing; and self-rated health: 0.3% missing.
      249 (49.4)255 (53.2)122 (43.9)128 (49.0)127 (56.2)127 (58.3)
      Prior ACP documentation148 (29.3)121 (25.2)84 (30.1)77 (29.4)64 (28.3)44 (20.1)
      AD = advance directive; ACP = advance care planning; SD = standard deviation.
      a Missing values: limited health literacy: 1.1% missing; finances: 1.6% missing; and self-rated health: 0.3% missing.
      Average ACP Behavior Change Survey scores increased more over time in the PREPARE vs. the AD-only arms (intervention group-by-time interaction, P < 0.001) among all survey versions (Fig. 1), demonstrating that all surveys can detect change in response to the ACP interventions. The results were similar when stratified by English and Spanish speakers (Appendix I and II).
      Figure thumbnail gr1
      Fig. 1Progressively shorter advance care planning engagement survey versions are able to detect change at one week, three months, six months, and 12 months by study arm.ab aEnglish and Spanish speakers had similar results as shown in and . bP-values in the plots reflect overall intervention group-by-time interactions.
      Within-group effect size estimates were larger in the PREPARE vs. the AD-only arms at each follow-up time point for all survey versions (Table 2). Intraclass correlation coefficients of the mean change scores between the original 82-item survey with progressively shorter versions were medium to high at all follow-up time points for both study arms (range over the four time points 0.78–0.97 for PREPARE and 0.76–0.98 for AD-only, all P < 0.001; Table 2). The 55- and 34-item versions had slightly lower mean change scores compared with the 82-item version, whereas the 15- and 9-item versions were variable, and the four-item version had slightly higher values. For example, in the PREPARE arm, the within-group mean change scores with standard deviation (SD) at 12 months compared with baseline were 0.82 (.9) for the 82-item survey, 0.73 (.9) for the 55-item survey, 0.68 (.9) for the 34-item survey, 0.70 (1.0) for the 15-item survey, 0.75 (1.0) for the 9-item survey, and 0.91 (1.3) for the 4-item survey (Table 2). However, these differences between the 82-item version and shorter surveys were all small and never exceeded 0.23 SD scale for PREPARE arm and 0.12 SD scale for AD-only arm across all time points (Table 2). Within-group results were similar for English- and Spanish-speaking participants (Appendix III and IV).
      Table 2Within-Group Effect Sizes and Correlation of Mean Change Scores Over Time Using Progressively Shorter ACP Engagement Survey Versions
      English and Spanish speakers had similar results as shown in Appendix III and IV.
      PREPARE (n = 481)Baseline to One-Week Follow-Up (n = 389)
      Missing values because of loss to follow-up at various follow-up time points.
      Baseline to Three-Month Follow-Up (n = 381)
      Missing values because of loss to follow-up at various follow-up time points.
      Baseline to Six-Month Follow-Up (n = 382)
      Missing values because of loss to follow-up at various follow-up time points.
      Baseline to 12-Month Follow-Up (n = 391)
      Missing values because of loss to follow-up at various follow-up time points.
      Effect SizeMean Change, Mean (SD)ICC
      All P-values for ICC were <0.001.
      (95% CI) of Mean Change
      Effect SizeMean Change, Mean (SD)ICC
      All P-values for ICC were <0.001.
      (95% CI) of Mean Change
      Effect SizeMean Change, Mean (SD)ICC
      All P-values for ICC were <0.001.
      (95% CI) of Mean Change
      Effect SizeMean Change, Mean (SD)ICC
      All P-values for ICC were <0.001.
      (95% CI) of Mean Change
      82-Item0.62.51 (.7)0.82.61 (.8)0.91.70 (.9)1.05.82 (.9)
      55-Item0.57.48 (.7)0.97 (0.96, 0.97)0.73.55 (.8)0.97 (0.96, 0.98)0.82.64 (.8)0.97 (0.97, 0.98)0.92.73 (.9)0.97 (0.96, 0.98)
      34-Item0.48.46 (.7)0.95 (0.94, 0.96)0.60.50 (.8)0.95 (0.94, 0.96)0.66.56 (.9)0.95 (0.94, 0.96)0.77.68 (.9)0.95 (0.94, 0.96)
      15-Item0.53.51 (.8)0.91 (0.88, 0.92)0.60.51 (.9)0.91 (0.89, 0.93)0.66.58 (.9)0.92 (0.90, 0.93)0.76.70 (1.0)0.92 (0.90, 0.93)
      9-Item0.54.56 (.9)0.87 (0.84, 0.89)0.61.56 (.9)0.87 (0.85, 0.90)0.66.63 (1.0)0.90 (0.88, 0.92)0.77.75 (1.0)0.89 (0.87, 0.91)
      4-Item0.60.67 (1.1)0.78 (0.73, 0.82)0.66.68 (1.1)0.79 (0.74, 0.83)0.74.76 (1.2)0.81 (0.77, 0.85)0.85.91 (1.3)0.81 (0.77, 0.84)
      AD-Only (n = 505)Baseline to One-Week Follow-Up (n = 440)
      Missing values because of loss to follow-up at various follow-up time points.
      Baseline to Three-Month Follow-Up (n = 416)
      Missing values because of loss to follow-up at various follow-up time points.
      Baseline to Six-Month Follow-Up (n = 434)
      Missing values because of loss to follow-up at various follow-up time points.
      Baseline to 12-Month Follow-Up (n = 440)
      Missing values because of loss to follow-up at various follow-up time points.
      Effect SizeMean Change, Mean (SD)ICC
      All P-values for ICC were <0.001.
      (95% CI) of Mean Change
      Effect SizeMean Change, Mean (SD)ICC
      All P-values for ICC were <0.001.
      (95% CI) of Mean Change
      Effect SizeMean Change, Mean (SD)ICC
      All P-values for ICC were <0.001.
      (95% CI) of Mean Change
      Effect SizeMean Change, Mean (SD)ICC
      All P-values for ICC were <0.001.
      (95% CI) of Mean Change
      82-Item0.28.22 (.6)0.42.34 (.7)0.49.41 (.8)0.64.52 (.9)
      55-Item0.27.21 (.6)0.97 (0.96, 0.97)0.41.32 (.7)0.97 (0.96, 0.97)0.47.38 (.8)0.97 (0.96, 0.98)0.59.48 (.9)0.98 (0.97, 0.98)
      34-Item0.24.20 (.7)0.94 (0.93, 0.95)0.35.29 (.7)0.95 (0.94, 0.96)0.40.33 (.8)0.95 (0.94, 0.96)0.51.43 (.9)0.96 (0.95, 0.97)
      15-Item0.20.18 (.8)0.90 (0.88, 0.92)0.33.29 (.8)0.91 (0.89, 0.93)0.35.32 (.9)0.90 (0.88, 0.92)0.44.41 (1.0)0.93 (0.91, 0.94)
      9-Item0.23.21 (.8)0.84 (0.81, 0.87)0.33.30 (.9)0.87 (0.84, 0.89)0.36.35 (.9)0.87 (0.84, 0.89)0.45.44 (1.0)0.90 (0.88, 0.92)
      4-Item0.26.27 (1.0)0.76 (0.71, 0.80)0.35.37 (1.0)0.79 (0.75, 0.83)0.38.41 (1.1)0.81 (0.77, 0.84)0.47.51 (1.2)0.84 (0.81, 0.87)
      ACP = advance care planning; ICC = intraclass correlation coefficient; AD = advance directive; SD = standard deviation.
      a English and Spanish speakers had similar results as shown in Appendix III and IV.
      b Missing values because of loss to follow-up at various follow-up time points.
      c All P-values for ICC were <0.001.
      Between-group effect size estimates for PREPARE vs. AD-only arms were very similar for all versions of the survey at all follow-up time points (Table 3; 82-item between-group effect size estimate ranges during the four periods: 0.24–0.31; 55-item [0.21–0.26]; 34-item [0.21–0.24]; 15-item [0.20–0.25]; 9-item [0.20–0.24]; and 4-item [0.23–0.29]). As observed for within-group estimates, the 55-item and 34-item versions had slightly lower between-group mean change differences compared with the 82-item version, whereas results were mixed with the 15- and 9-item versions, and the four-item version had slightly higher between-group differences. For example, for PREPARE vs. AD-only arms, the between-group differences of mean change scores with SD at 12 months compared with baseline were 0.30 (.9) for the 82-item survey, 0.25 (.9) for the 55-item survey, 0.25 (.9) for the 34-item survey, 0.29 (1.0) for the 15-item survey, 0.32 (1.0) for the 9-item survey, and 0.40 (1.2) for the 4-item survey (Table 3). However, these differences of mean change scores between the 82-item version and shorter surveys were all small and never exceeded 0.17 SD scale across all time points. Results for between-group comparisons of mean change score differences were similar for English- and Spanish-speaking participants (Appendix V and VI).
      Table 3Between-Group Effect Sizes and Differences of Mean Change Scores Over Time Using Progressively Shorter ACP Engagement Survey Versions
      English and Spanish speakers had similar results as shown in Appendix V and VI.
      PREPARE vs. AD-OnlyBaseline to One-Week Follow-UpBaseline to Three-Month Follow-UpBaseline to Six-Month Follow-UpBaseline to 12-Month Follow-Up
      Effect Size of DifferencesDifferences of Mean Change, Mean (SD)
      t-tests for comparing differences of mean change between progressively shorter survey versions and the original 82-item version had nonsignificant P-values with Bonferroni adjustment for multiple comparisons at a significance level of 0.05, indicating no obvious differences among survey versions.
      Effect Size of DifferencesDifferences of Mean Change, Mean (SD)
      t-tests for comparing differences of mean change between progressively shorter survey versions and the original 82-item version had nonsignificant P-values with Bonferroni adjustment for multiple comparisons at a significance level of 0.05, indicating no obvious differences among survey versions.
      Effect Size of DifferencesDifferences of Mean Change, Mean (SD)
      t-tests for comparing differences of mean change between progressively shorter survey versions and the original 82-item version had nonsignificant P-values with Bonferroni adjustment for multiple comparisons at a significance level of 0.05, indicating no obvious differences among survey versions.
      Effect Size of DifferencesDifferences of Mean Change, Mean (SD)
      t-tests for comparing differences of mean change between progressively shorter survey versions and the original 82-item version had nonsignificant P-values with Bonferroni adjustment for multiple comparisons at a significance level of 0.05, indicating no obvious differences among survey versions.
      82-Item0.24.28 (.7)0.29.27 (.8)0.31.29 (.8)0.30.30 (.9)
      55-Item0.21.27 (.6)0.24.23 (.7)0.26.26 (.8)0.24.25 (.9)
      34-Item0.21.26 (.7)0.22.21 (.8)0.23.23 (.8)0.24.25 (.9)
      15-Item0.25.33 (.8)0.20.22 (.8)0.24.26 (.9)0.25.29 (1.0)
      9-Item0.24.36 (.9)0.20.26 (.9)0.22.28 (.9)0.24.32 (1.0)
      4-Item0.25.40 (1.0)0.23.31 (1.1)0.26.35 (1.1)0.29.40 (1.2)
      ACP = advance care planning; AD = advance directive; SD = standard deviation.
      a English and Spanish speakers had similar results as shown in Appendix V and VI.
      b t-tests for comparing differences of mean change between progressively shorter survey versions and the original 82-item version had nonsignificant P-values with Bonferroni adjustment for multiple comparisons at a significance level of 0.05, indicating no obvious differences among survey versions.

      Discussion

      Using randomized clinical trial data, with four follow-up time points among a large cohort of older adults, we demonstrated that all survey versions were able to detect change in a broad range of ACP behaviors over time in response to ACP interventions. The surveys worked well among both English- and Spanish-speaking participants, although Spanish speakers had higher rates of limited health literacy and were more likely to have less than a high school education. Having several psychometrically sound shortened versions of the ACP Engagement Survey provides flexibility for research and quality improvement initiatives when choosing surveys to measure the effectiveness of ACP programs.
      We found that the original 82-item version of the ACP Engagement Survey and five progressively shorter versions (i.e., 55-item, 34-item, 15-item, 9-item, and 4-item) can reliably detect both within-group and between-group differences for ACP interventions over all time points (i.e., one week, three months, six months, and 12 months). Both within-group and between-group effect sizes tended to be higher using the full 82-item survey, suggesting this version may be most appropriate when maximum power is required, for example, for small studies. However, the shorter versions of the survey were all able to detect both within-group and between-group changes, suggesting that they are acceptable alternatives in most clinical and research settings.
      This study allowed us to quantify a clinically meaningful change in ACP Engagement Survey scores based on effect sizes using standard thresholds.
      • Cohen J.
      Statistical power analysis for the behavioral sciences.
      Small effect sizes (0.20–0.49) were associated with mean change scores of approximately 0.2–0.3 points. Moderate effect sizes (0.50–0.79) were associated with mean change scores of approximately 0.4–0.5 points. Large effect sizes (≥0.80) were associated with mean change scores of ≥0.6 points. Therefore, the smallest clinically meaningful change in response to an ACP intervention would be approximately 0.2 points and is an evidence that patients are moving along the behavior change pathway—from precontemplation, to contemplation, to preparation, to action. Larger changes of 0.6 or greater likely reflect ACP actions that are farther down the behavior change pathway. For example, in a prior validation study of the survey in 559 respondents in two countries, a score change of 1.0 was associated with having completed a prior AD.
      • Sudore R.L.
      • Heyland D.K.
      • Barnes D.E.
      • et al.
      Measuring advance care planning: optimizing the advance care planning engagement survey.
      This study also provided detailed within-group and between-group effects size information for each version of the survey at multiple follow-up time points compared with baseline for the overall cohort as well as for English and Spanish speakers. These results are important because it will allow ACP researchers to calculate power and estimate sample sizes for future clinical trials. Choice of the survey version may be based on the length of the survey desired to reduce response burden, the ACP information important to the research question (as the Behavior Change process questions concerning contemplation, flexibility in surrogate decision making, and asking doctors were the most often deleted questions from shorter versions), and the follow-up time proposed (i.e., one week, three months, six months, or 12 months).
      The strengths of this study include the rigorous and systematic validation of all survey versions, assessment of the survey's ability to detect change over time in response to interventions in English and Spanish speakers and use of published trial data. This study does have some limitations. Generalizability may be limited because the validation only took place in one San Francisco health delivery system, with a predominance of older adults. However, the primary care sample was racially and ethnically diverse. Although inclusion criteria required chronic or serious illness for the trial, we do not know whether the results would be similar among patients from specialty clinics or patients who speak a language other than English or Spanish. Future studies will also need to assess the ability of shorter survey versions to detect change in response to different ACP interventions in varying patient populations and whether the survey can be used to help tailor ACP discussions based on readiness and behaviors that have not yet been completed in the clinical setting.
      In conclusion, progressively shorter versions of the ACP Engagement Survey, including a four-item version, are psychometrically sound and able to efficiently and effectively measure change in ACP behaviors in response to ACP interventions. The choice of which survey version to use will depend on overall data collection burden, available resources, and the desire to look at survey subscales or specific survey domains.

      Disclosures and Acknowledgments

      Although this study was nonfunded, the original Veterans Affairs trial was funded by the U.S. Department of Veterans Affairs. Dr. Sudore is also supported in part by a National Institutes of Health, National Institutes on Aging K24AG054415 award. The funding sources had no role in the design, conduct, or analysis of this study or in the decision to submit the article for publication. The authors report no conflicts of interest related to the work described in this article. The corresponding author, Dr. Shi, had full access to all the data in the study and takes responsibility for the integrity of the data and accuracy of the data analysis.

      Appendix

      Figure thumbnail fx1
      Appendix IEnglish-speaking participants: advance care planning engagement scores at baseline, one week, three months, six months, and 12 months for progressively shorter survey versions by study arma. aP-values in the plots reflect overall intervention group-by-time interactions.
      Figure thumbnail fx2
      Appendix IISpanish-speaking participants: advance care planning engagement scores at baseline, one week, three months, six months, and 12 months for progressively shorter survey versions by study arma. aP-values in the plots reflect overall intervention group-by-time interactions.
      Appendix IIIEnglish-Speaking Participants: Within-Group Effect Sizes and Correlation of Mean Change Scores Over Time Using Progressively Shorter ACP Engagement Survey Versions
      PREPARE (n = 262)Baseline to One-Week Follow-Up (n = 219)
      Missing values because of loss-to-follow up at various follow-up time points.
      Baseline to Three-Month Follow-Up (n = 218)
      Missing values because of loss-to-follow up at various follow-up time points.
      Baseline to Six-Month Follow-Up (n = 210)
      Missing values because of loss-to-follow up at various follow-up time points.
      Baseline to 12-Month Follow-Up (n = 210)
      Missing values because of loss-to-follow up at various follow-up time points.
      Effect SizeMean Change, Mean (SD)ICC
      All P-values for ICC are <0.001.
      (95% CI) of Mean Change
      Effect SizeMean Change, Mean (SD)ICC
      All P-values for ICC are <0.001.
      (95% CI) of Mean Change
      Effect SizeMean Change, Mean (SD)ICC
      All P-values for ICC are <0.001.
      (95% CI) of Mean Change
      Effect SizeMean Change, Mean (SD)ICC
      All P-values for ICC are <0.001.
      (95% CI) of Mean Change
      82-Item0.57.47 (.7)0.71.55 (.7)0.86.66 (.9)0.95.76 (.8)
      55-Item0.52.44 (.6)0.96 (0.95, 0.97)0.62.48 (.7)0.96 (0.95, 0.97)0.76.58 (.8)0.97 (0.96, 0.98)0.82.65 (.7)0.96 (0.95, 0.97)
      34-Item0.42.40 (.7)0.94 (0.92, 0.95)0.47.40 (.7)0.94 (0.92, 0.95)0.56.47 (.8)0.95 (0.93, 0.96)0.66.57 (.8)0.94 (0.92, 0.95)
      15-Item0.45.45 (.8)0.91 (0.88, 0.93)0.46.42 (.8)0.90 (0.87, 0.93)0.56.49 (.9)0.91 (0.89, 0.93)0.64.58 (.9)0.90 (0.86, 0.92)
      9-Item0.47.49 (.8)0.87 (0.83, 0.90)0.50.48 (.8)0.85 (0.81, 0.89)0.59.57 (.9)0.89 (0.86, 0.92)0.66.65 (.9)0.86 (0.82, 0.90)
      4-Item0.54.60 (1.0)0.78 (0.72, 0.83)0.55.58 (1.1)0.78 (0.72, 0.83)0.67.71 (1.2)0.83 (0.78, 0.87)0.73.79 (1.2)0.80 (0.74, 0.85)
      AD-Only (n = 279)Baseline to One-Week Follow-Up (n = 242)
      Missing values because of loss-to-follow up at various follow-up time points.
      Baseline to Three-Month Follow-Up (n = 234)
      Missing values because of loss-to-follow up at various follow-up time points.
      Baseline to Six-Month Follow-Up (n = 247)
      Missing values because of loss-to-follow up at various follow-up time points.
      Baseline to 12-Month Follow-Up (n = 243)
      Missing values because of loss-to-follow up at various follow-up time points.
      Effect SizeMean Change, Mean (SD)ICC
      All P-values for ICC are <0.001.
      (95% CI) of Mean Change
      Effect SizeMean Change, Mean (SD)ICC
      All P-values for ICC are <0.001.
      (95% CI) of Mean Change
      Effect SizeMean Change, Mean (SD)ICC
      All P-values for ICC are <0.001.
      (95% CI) of Mean Change
      Effect SizeMean Change, Mean (SD)ICC
      All P-values for ICC are <0.001.
      (95% CI) of Mean Change
      82-Item0.34.24 (.6)0.41.33 (.7)0.49.40 (.8)0.62.52 (.8)
      55-Item0.34.24 (.6)0.97 (0.96, 0.97)0.43.33 (.6)0.97 (0.96, 0.97)0.48.37 (.7)0.97 (0.96, 0.97)0.61.48 (.8)0.97 (0.96, 0.98)
      34-Item0.30.22 (.7)0.95 (0.93, 0.96)0.34.27 (.7)0.94 (0.92, 0.95)0.37.30 (.8)0.94 (0.93, 0.96)0.48.41 (.8)0.94 (0.93, 0.96)
      15-Item0.22.16 (.7)0.91 (0.88, 0.93)0.29.25 (.8)0.91 (0.88, 0.93)0.30.26 (.8)0.91 (0.88, 0.93)0.40.36 (.9)0.92 (0.89, 0.94)
      9-Item0.25.22 (.8)0.86 (0.82, 0.89)0.30.28 (.8)0.87 (0.83, 0.90)0.33.32 (.9)0.87 (0.84, 0.90)0.42.43 (1.0)0.87 (0.84, 0.90)
      4-Item0.26.27 (1.0)0.77 (0.70, 0.82)0.27.31 (1.0)0.78 (0.72, 0.83)0.32.35 (1.1)0.82 (0.76, 0.86)0.42.50 (1.2)0.82 (0.77, 0.86)
      ACP = advance care planning; ICC = intraclass correlation coefficient; AD = advance directive; SD = standard deviation.
      a Missing values because of loss-to-follow up at various follow-up time points.
      b All P-values for ICC are <0.001.
      Appendix IVSpanish-Speaking Participants: Within-Group Effect Sizes and Correlation of Mean Change Scores Over Time Using Progressively Shorter ACP Engagement Survey Versions
      PREPARE (n = 219)Baseline to One-Week Follow-Up (n = 170)
      Missing values because of loss-to-follow up at various follow-up time points.
      Baseline to Three-Month Follow-Up (n = 163)
      Missing values because of loss-to-follow up at various follow-up time points.
      Baseline to Six-Month Follow-Up (n = 172)
      Missing values because of loss-to-follow up at various follow-up time points.
      Baseline to 12-Month Follow-Up (n = 181)
      Missing values because of loss-to-follow up at various follow-up time points.
      Effect SizeMean Change, Mean (SD)ICC
      All P-values for ICC are <0.001.
      (95% CI) of Mean Change
      Effect SizeMean Change, Mean (SD)ICC
      All P-values for ICC are <0.001.
      (95% CI) of Mean Change
      Effect SizeMean Change, Mean (SD)ICC
      All P-values for ICC are <0.001.
      (95% CI) of Mean Change
      Effect SizeMean Change, Mean (SD)ICC
      All P-values for ICC are <0.001.
      (95% CI) of Mean Change
      82-Item0.70.56 (.7)0.97.69 (.9)1.03.74 (.9)1.25.89 (1.0)
      55-Item0.69.54 (.7)0.97 (0.96, 0.98)0.93.65 (.9)0.98 (0.97, 0.98).71 (.9)0.98 (0.97, 0.98)1.16.83 (1.0)0.98 (0.97, 0.98)
      34-Item0.62.53 (.8)0.95 (0.94, 0.97)0.83.64 (.9)0.96 (0.95, 0.97)0.86.67 (.9)0.96 (0.95, 0.97)1.02.81 (1.0)0.97 (0.95, 0.97)
      15-Item0.63.59 (.8)0.90 (0.87, 0.93)0.78.64 (.9)0.92 (0.90, 0.94)0.84.68 (1.0)0.92 (0.90, 0.94)0.99.84 (1.0)0.93 (0.91, 0.95)
      9-Item0.70.65 (.9)0.87 (0.82, 0.90)0.80.67 (1.0)0.89 (0.86, 0.92)0.83.70 (1.0)0.91 (0.88, 0.93)1.01.88 (1.1)0.91 (0.89, 0.94)
      4-Item0.71.75 (1.2)0.77 (0.69, 0.83)0.84.81 (1.1)0.79 (0.71, 0.85)0.85.83 (1.3)0.79 (0.72, 0.84)1.071.05 (1.3)0.82 (0.75, 0.86)
      AD-Only (n = 226)Baseline to One-Week Follow-Up (n = 198)
      Missing values because of loss-to-follow up at various follow-up time points.
      Baseline to Three-Month Follow-Up (n = 182)
      Missing values because of loss-to-follow up at various follow-up time points.
      Baseline to Six-Month Follow-Up (n = 187)
      Missing values because of loss-to-follow up at various follow-up time points.
      Baseline to 12-Month Follow-Up (n = 197)
      Missing values because of loss-to-follow up at various follow-up time points.
      Effect SizeMean Change, Mean (SD)ICC
      All P-values for ICC are <0.001.
      (95% CI) of Mean Change
      Effect SizeMean Change, Mean (SD)ICC
      All P-values for ICC are <0.001.
      (95% CI) of Mean Change
      Effect SizeMean Change, Mean (SD)ICC
      All P-values for ICC are <0.001.
      (95% CI) of Mean Change
      Effect SizeMean Change, Mean (SD)ICC
      All P-values for ICC are <0.001.
      (95% CI) of Mean Change
      82-Item0.24.20 (.6)0.47.35 (.7)0.54.41 (.8)0.72.53 (1.0)
      55-Item0.21.18 (.6)0.97 (0.96, 0.98)0.41.32 (.7)0.97 (0.96, 0.98)0.50.39 (.8)0.98 (0.97, 0.98)0.65.48 (1.0)0.98 (0.97, 0.99)
      34-Item0.20.17 (.7)0.94 (0.92, 0.96)0.40.31 (.8)0.95 (0.94, 0.97)0.45.37 (.8)0.96 (0.94, 0.97)0.58.45 (1.0)0.98 (0.97, 0.98)
      15-Item0.21.20 (.8)0.89 (0.86, 0.92)0.40.34 (.8)0.92 (0.89, 0.94)0.43.39 (.9)0.90 (0.86, 0.92)0.55.47 (1.0)0.94 (0.92, 0.96)
      9-Item0.20.19 (.9)0.82 (0.77, 0.87)0.39.33 (.9)0.87 (0.82, 0.90)0.42.39 (.9)0.87 (0.82, 0.90)0.51.45 (1.1)0.92 (0.90, 0.94)
      4-Item0.26.27 (1.0)0.74 (0.66, 0.80)0.46.44 (1.1)0.80 (0.74, 0.85)0.49.49 (1.0)0.81 (0.74, 0.86)0.56.52 (1.2)0.86 (0.82, 0.90)
      ACP = advance care planning; ICC = intraclass correlation coefficient; AD = advance directive; SD = standard deviation.
      a Missing values because of loss-to-follow up at various follow-up time points.
      b All P-values for ICC are <0.001.
      Appendix VEnglish-Speaking Participants: Between-Group Effect Sizes and Differences of Mean Change Scores Over Time Using Progressively Shorter ACP Engagement Survey Versions
      PREPARE vs. AD-OnlyBaseline to One-Week Follow-UpBaseline to Three-Month Follow-UpBaseline to Six-Month Follow-UpBaseline to 12-Month Follow-Up
      Effect Size of DifferencesDifferences of Mean Change, Mean (SD)
      t-tests for comparing differences of mean change between progressively shorter survey versions and the original 82-item version had nonsignificant P-values with Bonferroni adjustment for multiple comparisons at a significance level of 0.05, which meant no obvious differences among survey versions.
      Effect Size of DifferencesDifferences of Mean Change, Mean (SD)
      t-tests for comparing differences of mean change between progressively shorter survey versions and the original 82-item version had nonsignificant P-values with Bonferroni adjustment for multiple comparisons at a significance level of 0.05, which meant no obvious differences among survey versions.
      Effect Size of DifferencesDifferences of Mean Change, Mean (SD)
      t-tests for comparing differences of mean change between progressively shorter survey versions and the original 82-item version had nonsignificant P-values with Bonferroni adjustment for multiple comparisons at a significance level of 0.05, which meant no obvious differences among survey versions.
      Effect Size of DifferencesDifferences of Mean Change, Mean (SD)
      t-tests for comparing differences of mean change between progressively shorter survey versions and the original 82-item version had nonsignificant P-values with Bonferroni adjustment for multiple comparisons at a significance level of 0.05, which meant no obvious differences among survey versions.
      82-Item0.13.23 (.7)0.20.22 (.7)0.26.26 (.8)0.23.25 (.8)
      55-Item0.12.20 (.6)0.13.16 (.7)0.22.21 (.8)0.15.17 (.8)
      34-Item0.11.18 (.7)0.12.12 (.7)0.19.16 (.8)0.18.16 (.8)
      15-Item0.18.28 (.8)0.12.17 (.8)0.20.23 (.9)0.19.22 (.9)
      9-Item0.14.28 (.8)0.12.20 (.8)0.18.25 (.9)0.16.22 (1.0)
      4-Item0.15.33 (1.0)0.16.26 (1.0)0.23.36 (1.1)0.18.29 (1.2)
      ACP = advance care planning; AD = advance directive; SD = standard deviation.
      a t-tests for comparing differences of mean change between progressively shorter survey versions and the original 82-item version had nonsignificant P-values with Bonferroni adjustment for multiple comparisons at a significance level of 0.05, which meant no obvious differences among survey versions.
      Appendix VISpanish-Speaking Participants: Between-Group Effect Sizes and Differences of Mean Change Scores Over Time Using Progressively Shorter ACP Engagement Survey Versions
      PREPARE vs. AD-OnlyBaseline to One-Week Follow-UpBaseline to Three-Month Follow-UpBaseline to Six-Month Follow-UpBaseline to 12-Month Follow-Up
      Effect Size of DifferencesDifferences of Mean Change, Mean (SD)
      t-tests for comparing differences of mean change between progressively shorter survey versions and the original 82-item version had nonsignificant P-values with Bonferroni adjustment for multiple comparisons at a significance level of 0.05, which meant no obvious differences among survey versions.
      Effect Size of DifferencesDifferences of Mean Change, Mean (SD)
      t-tests for comparing differences of mean change between progressively shorter survey versions and the original 82-item version had nonsignificant P-values with Bonferroni adjustment for multiple comparisons at a significance level of 0.05, which meant no obvious differences among survey versions.
      Effect Size of DifferencesDifferences of Mean Change, Mean (SD)
      t-tests for comparing differences of mean change between progressively shorter survey versions and the original 82-item version had nonsignificant P-values with Bonferroni adjustment for multiple comparisons at a significance level of 0.05, which meant no obvious differences among survey versions.
      Effect Size of DifferencesDifferences of Mean Change, Mean (SD)
      t-tests for comparing differences of mean change between progressively shorter survey versions and the original 82-item version had nonsignificant P-values with Bonferroni adjustment for multiple comparisons at a significance level of 0.05, which meant no obvious differences among survey versions.
      82-Item0.37.36 (.7)0.40.33 (.8)0.37.32 (.9)0.37.37 (1.0)
      55-Item0.35.36 (.7)0.36.33 (.8)0.33.32 (.8)0.36.35 (1.0)
      34-Item0.34.36 (.7)0.30.32 (.8)0.32.30 (.9)0.35.36 (1.0)
      15-Item0.39.39 (.8)0.31.30 (.9)0.29.30 (.9)0.38.37 (1.0)
      9-Item0.39.46 (.9)0.31.34 (.9)0.29.31 (1.0)0.38.43 (1.1)
      4-Item0.40.48 (1.1)0.34.38 (1.1)0.32.34 (1.1)0.47.52 (1.2)
      ACP = advance care planning; AD = advance directive; SD = standard deviation.
      a t-tests for comparing differences of mean change between progressively shorter survey versions and the original 82-item version had nonsignificant P-values with Bonferroni adjustment for multiple comparisons at a significance level of 0.05, which meant no obvious differences among survey versions.

      References

        • Klingler C.
        • In der Schmitten J.
        • Marckmann G.
        Does facilitated advance care planning reduce the costs of care near the end of life? Systematic review and ethical considerations.
        Palliat Med. 2016; 30: 423-433
        • Silveira M.J.
        • Kim S.Y.
        • Langa K.M.
        Advance directives and outcomes of surrogate decision making before death.
        N Engl J Med. 2010; 362: 1211-1218
        • Hammes B.J.
        • Rooney B.L.
        • Gundrum J.D.
        A comparative, retrospective, observational study of the prevalence, availability, and specificity of advance care plans in a county that implemented an advance care planning microsystem.
        J Am Geriatr Soc. 2010; 58: 1249-1255
        • Detering K.M.
        • Hancock A.D.
        • Reade M.C.
        • Silvester W.
        The impact of advance care planning on end of life care in elderly patients: randomised controlled trial.
        BMJ. 2010; 340: c1345
        • Pope T.M.
        Legal briefing: medicare coverage of advance care planning.
        J Clin Ethics. 2015; 26: 361-367
        • Sudore R.L.
        • Fried T.R.
        Redefining the ‘‘planning” in advance care planning: preparing for end-of-life decision making.
        Ann Intern Med. 2010; 153: 256-261
        • Sudore R.L.
        • Schickedanz A.D.
        • Landefeld C.S.
        • et al.
        Engagement in multiple steps of the advance care planning process: a descriptive study of diverse older adults.
        J Am Geriatr Soc. 2008; 56: 1006-1013
        • Fried T.R.
        • Redding C.A.
        • Robbins M.L.
        • et al.
        Stages of change for the component behaviors of advance care planning.
        J Am Geriatr Soc. 2010; 58: 2329-2336
        • Fried T.R.
        • Redding C.A.
        • Robbins M.L.
        • et al.
        Promoting advance care planning as health behavior change: development of scales to assess decisional balance, medical and religious beliefs, and processes of change.
        Patient Educ Couns. 2012; 86: 25-32
        • Sudore R.L.
        • Heyland D.K.
        • Lum H.D.
        • et al.
        Outcomes that define successful advance care planning: a Delphi panel consensus.
        J Pain Symptom Manage. 2018; 55: 245-255
        • Sudore R.L.
        • Knight S.J.
        • McMahan R.D.
        • et al.
        A novel website to prepare diverse older adults for decision making and advance care planning: a pilot study.
        J Pain Symptom Manage. 2014; 47: 674-686
        • Lum H.D.
        • Barnes D.E.
        • Katen M.T.
        • et al.
        Improving a full range of advance care planning behavior change and action domains: the PREPARE randomized trial.
        J Pain Symptom Manage. 2018; 56: 575-581
        • Sudore R.L.
        • Stewart A.L.
        • Knight S.J.
        • et al.
        Development and validation of a questionnaire to detect behavior change in multiple advance care planning behaviors.
        PLoS One. 2013; 8: e72465
        • Sudore R.L.
        • Heyland D.K.
        • Barnes D.E.
        • et al.
        Measuring advance care planning: optimizing the advance care planning engagement survey.
        J Pain Symptom Manage. 2017; 53: 669-681
        • Howard M.
        • Bonham A.J.
        • Heyland D.K.
        • et al.
        Measuring engagement in advance care planning: a cross-sectional multicentre feasibility study.
        BMJ Open. 2016; 6: e010375
        • Sudore R.L.
        • Schillinger D.
        • Katen M.T.
        • et al.
        Engaging diverse English- and Spanish-speaking older adults in advance care planning: the PREPARE randomized clinical trial.
        JAMA Intern Med. 2018; 178: 1616-1625
        • Sudore R.L.
        • Barnes D.E.
        • Le G.M.
        • et al.
        Improving advance care planning for English-speaking and Spanish-speaking older adults: study protocol for the PREPARE randomized controlled trial.
        BMJ Open. 2016; 6: e011705
        • Chew L.D.
        • Griffin J.M.
        • Partin M.R.
        • et al.
        Validation of screening questions for limited health literacy in a large VA outpatient population.
        J Gen Intern Med. 2008; 23: 561-566
        • Cohen J.
        Statistical power analysis for the behavioral sciences.
        2nd ed. Lawrence Erlbaum Associates, Hillsdale, NJ1988