If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Address correspondence to: Rebecca L. Sudore, MD, University of California, San Francisco, School of Medicine, SFVAMC 4150 Clement Street, #151R, San Francisco, CA 94121, USA.
Division of Geriatrics, Department of Medicine, University of California, San Francisco, San Francisco, California, USASan Francisco Veterans Affairs Medical Center, San Francisco, California, USA
Clinical Evaluation Research Unit, Kingston General Hospital, Kingston, Ontario, CanadaDepartment of Public Health, Queen's University, Kingston, Ontario, Canada
San Francisco Veterans Affairs Medical Center, San Francisco, California, USADepartment of Psychiatry, University of California, San Francisco, San Francisco, California, USADepartment of Epidemiology and Biostatistics, University of California, San Francisco, San Francisco, California, USATideswell at UCSF, San Francisco, California, USA
Division of Geriatrics, Department of Medicine, University of California, San Francisco, San Francisco, California, USADepartment of Epidemiology and Biostatistics, University of California, San Francisco, San Francisco, California, USA
Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Ontario, CanadaDepartment of Medicine, McMaster University, Hamilton, Ontario, Canada
A validated 82-item Advance Care Planning (ACP) Engagement Survey measures a broad range of behaviors. However, concise surveys are needed.
Objectives
The objective of this study was to validate shorter versions of the survey.
Methods
The survey included 57 process (e.g., readiness) and 25 action items (e.g., discussions). For item reduction, we systematically eliminated questions based on face validity, item nonresponse, redundancy, ceiling effects, and factor analysis. We assessed internal consistency (Cronbach's alpha) and construct validity with cross-sectional correlations and the ability of the progressively shorter survey versions to detect change one week after exposure to an ACP intervention (Pearson correlation coefficients).
Results
Five hundred one participants (four Canadian and three US sites) were included in item reduction (mean age 69 years [±10], 41% nonwhite). Because of high correlations between readiness and action items, all action items were removed. Because of high correlations and ceiling effects, two process items were removed. Successive factor analysis then created 55-, 34-, 15-, nine-, and four-item versions; 664 participants (from three US ACP clinical trials) were included in validity analysis (age 65 years [±8], 72% nonwhite, 34% Spanish speaking). Cronbach's alphas were high for all versions (four items 0.84–55 items 0.97). Compared with the original survey, cross-sectional correlations were high (four items 0.85; 55 items 0.97) as were delta correlations (four items 0.68; 55 items 0.93).
Conclusion
Shorter versions of the ACP Engagement Survey are valid, internally consistent, and able to detect change across a broad range of ACP behaviors for English and Spanish speakers. Shorter ACP surveys can efficiently measure broad ACP behaviors in research and clinical settings.
Advance care planning (ACP) is a process that supports adults at any age or stage of health in understanding and sharing their personal values, life goals, and preferences regarding future medical care.
Increasing attention has been paid to ACP over the past several years by health care systems, insurers, and researchers because ACP can improve patient and family satisfaction, increase alignment of medical care with patients' values, and reduce costs.
A comparative, retrospective, observational study of the prevalence, availability, and specificity of advance care plans in a county that implemented an advance care planning microsystem.
With the proliferation of new ACP initiatives, tools, and reimbursement models for ACP, measuring the success of an ACP program has become increasingly important. Traditionally, successful ACP has been measured only by the completion of an advance directive. However, several studies have shown that the ACP process is complex and involves many different ACP behaviors, such as identifying one's values, choosing a surrogate decision maker, and discussing values with surrogates and clinicians and completing advance directives.
Promoting advance care planning as health behavior change: development of scales to assess decisional balance, medical and religious beliefs, and processes of change.
for widespread use in research and clinical settings. The purpose of this study was to create and validate progressively shorter versions of the survey that would be able to detect change in response to an ACP intervention.
Methods
The development and validation of the ACP Engagement Survey have been described in detail elsewhere.
The survey focuses on four behavior change constructs (i.e., knowledge, contemplation, self-efficacy, and readiness) within four ACP domains (i.e., surrogate decision makers, values and quality of life, leeway in surrogate decision making, and asking doctors questions). After pilot testing with patients and ongoing input from ACP experts, 33 questions were added to the original survey concerning desired medical treatment and the involvement of friends and family in ACP, as has been previously described.
This resulted in an 82-item version of the survey with 57 behavior change “Process Measures” (measured with five-point Likert response options) and 25 ACP “Action Measures” (measured with “yes” or “no” response options) (Table 1). The survey has good internal consistency (Process Measures Cronbach's alpha 0.94) and test-retest reliability (Process Measures intraclass correlation 0.70; Action Measures 0.87).
Response options for Process-Readiness items (five-point Likert responses): “I have never thought about it,” “I have thought about it, but I am not ready to do it,” “I am thinking about doing it in the next 6 months,” “I am definitely planning to do it in the next 30 days,” “I have already done it.”
Process
How ready are you to decide who you want your medical decision maker to be?
Have you signed official papers naming a person or group of people to make medical decisions for you?
20
Readiness
Process
How ready are you to sign official papers naming a person or group of people to make medical decisions for you?
X
X
X
X
X
Domain: Quality of life—health situations
21
Contemplation
Process
How much have you thought about whether or not certain health situations would make your life not worth living?
X
22
Contemplation
Process
How much have you thought about talking with your medical decision maker about whether or not certain health situations would make your life not worth living?
X
23
Contemplation
Process
How much have you thought about talking with your doctor about whether or not certain health situations would make your life not worth living?
X
24
Contemplation
Process
How much have you thought about talking with your other family and friends about whether or not certain health situations would make your life not worth living?
X
25
Self-efficacy
Process
How confident are you that today you could talk with your medical decision maker about whether or not certain health situations would make your life not worth living?
X
X
26
Self- Efficacy
Process
How confident are you that today you could talk with your doctor about whether or not certain health situations would make your life not worth living?
X
X
27
Self-efficacy
Process
How confident are you that today you could talk with your other family and friends about whether or not certain health situations would make your life not worth living?
Have you signed official papers to put your wishes in writing about whether or not certain health situations would make your life not worth living? These forms are sometimes called an advance directive or living will.
37
Readiness
Process
How ready are you to sign official papers putting your wishes in writing about whether or not certain health situations would make your life not worth living?
X
Domain: quality of life—medical care at the end of life
38
Contemplation
Process
How much have you thought about the care you would want if you were very sick or near the end of life?
X
39
Contemplation
Process
How much have you thought about talking with your medical decision maker about the care you would want if you were very sick or near the end of life?
X
40
Contemplation
Process
How much have you thought about talking with your doctors about the care you would want if you were very sick or near the end of life?
X
41
Contemplation
Process
How much have you thought about talking with your other family and friends about the care you would want if you were very sick or near the end of life?
X
X
42
Self- Efficacy
Process
How confident are you that today you could talk with your medical decision maker about the care you would want if you were very sick or near the end of life?
X
X
X
X
43
Self-efficacy
Process
How confident are you that today you could talk with your doctor about the care you would want if you were very sick or near the end of life?
X
X
X
X
44
Self-efficacy
Process
How confident are you that today you could talk with your other family and friends about the care you would want if you were very sick or near the end of life?
Have you ever talked with your other family and friends about what kind of medical care you want if you were very sick or near the end of life?
52
Readiness
Process
How ready are you to talk to your other family and friends about the kind of medical care you would want if you were very sick or near the end of life?
Have you signed official papers to put your wishes in writing about the kind of medical care you would want if you were very sick or near the end of life? These forms are sometimes called an advance directive or living will.
54
Readiness
Process
How ready are you to sign official papers putting your wishes in writing about the kind of medical care you would want if you were very sick or near the end of life?
X
X
X
X
X
Domain: flexibility for surrogate decision making
55
Knowledge
Process
How well informed are you about what it means to give a medical decision maker flexibility to make future decisions?
X
56
Knowledge
Process
How well informed are you about the different amounts of flexibility a person can give their medical decision maker?
X
57
Contemplation
Process
How much have you thought about the amount of flexibility you would want to give your medical decision maker?
X
58
Contemplation
Process
How much have you thought about talking with your medical decision maker about how much flexibility you want to give them?
X
X
59
Contemplation
Process
How much have you thought about talking with your doctor about how much flexibility you want to give your decision maker?
X
60
Contemplation
Process
How much have you thought about talking with other friends and family about how much flexibility you want to give your decision maker?
X
61
Self-efficacy
Process
How confident are you that today you could talk with your decision maker about how much flexibility you want to give them?
X
X
62
Self-efficacy
Process
How confident are you that today you could talk with your doctor about how much flexibility you want to give your medical decision maker?
X
X
X
63
Self-efficacy
Process
How confident are you that today you could talk with your other family and friends about how much flexibility you want to give your medical decision maker?
Have you ever asked your doctor to repeat information if you did not understand it the first time?
82
Readiness
Process
How ready are you to ask your doctor questions to help you make a good medical decision?
X
X
X
a Items reduced before factor analysis.
b Response options for Process-Knowledge items (five-point Likert responses): “Not at all,” “A little,” “Somewhat,” “Fairly,” “Extremely.”
c Response options for Process-Contemplation items (five-point Likert responses): “Never,” “Once,” “A few times,” “Several times,” “A lot.”
d Response options for Process-Self-efficacy items (five-point Likert responses): “Not at all,” “A little,” “Somewhat,” “Fairly,” “Extremely.”
e Response options for Process-Readiness items (five-point Likert responses): “I have never thought about it,” “I have thought about it, but I am not ready to do it,” “I am thinking about doing it in the next 6 months,” “I am definitely planning to do it in the next 30 days,” “I have already done it.”
f Response options for Action items: “Yes,” “no,” or “I am not sure.” A “no” or “I am not sure” response are both coded as “no.”
The present study had two phases. The goal of Phase 1 “Item Reduction” was to systematically reduce items and create progressively shorter versions of the survey (Fig. 1). The goal of Phase 2 “Internal Consistency and Construct Validity” was to test the internal consistency and the construct validity of the shorter versions and ensure they could detect change in response to an ACP intervention. Informed consent was obtained from all participants, and the study was approved by the respective Institutional Review Boards in the United States and Canada.
Fig. 1Methods for creating and validating shorter versions of the Advance Care Planning Engagement Survey.
For Phase 1, “Item Reduction,” we included participants from both the United States (US) and Canada. In the US, from February 2013 to September 2014, we included baseline data of English-speaking participants enrolled in three ongoing randomized trials of ACP interventions at the San Francisco Veterans Affairs (VA) Medical Center and San Francisco General Hospital (SFGH). The methods of these trials have been published.
In brief, participants were included if they were 60 years or older (VA) or 55 years or older (SFGH), had two or more chronic medical illnesses, and had seen a primary care physician two or more times in the past year. Patients were excluded if they had cognitive impairment or dementia, blindness, deafness, or psychosis determined from administrative data or screening.
In Canada, from April to September 2014, we included participants who reported being able to speak and read English and whose physicians reported they did not have cognitive impairment. A convenience sample of patients from varying age groups was recruited in the following clinical settings as part of ongoing additional studies and clinical activities: 1) primary care in Hamilton, Ontario and Edmonton, Alberta, including patients of 50 years or older, 2) inpatient hospital settings in Hamilton and Kingston, Ontario, including patients of 80 years or older or 55 years or older with clinical markers of advanced chronic disease, 3) outpatient cancer centers in Edmonton, Alberta; Hamilton, Ontario; and Kelowna and Vancouver, British Columbia, including patients of 19 years or older, and 4) outpatient dialysis care centers in Edmonton, Alberta, including patients 19 years or older.
In Phase 2, “Internal Consistency and Construct Validity,” we included participants from the US who were enrolled between February 2013 and March 2016 in three ongoing US ACP trials. Some of Phase 1 US participants were also included in Phase 2. However, because the validity analyses are descriptive and not predictive, we decided to include all trial participants enrolled at the time of the Phase 2 analysis. This included both English- and Spanish-speaking participants at the time of Phase 2 analysis. In the Phase 2, we included both baseline and one-week follow-up trial data to be able to assess the survey's ability to detect change in response to an ACP intervention.
Phase 1, Item Reduction
We conducted incremental analyses to create successively shorter survey versions (Fig. 1). We considered that longer versions would have superior psychometric properties and may be more appropriate for research, whereas shorter versions may have lower validity but may be more appropriate in the clinical setting.
First, we assessed survey items for face validity. Feedback was obtained from all authors and five research assistants who were involved in data collection and had obtained feedback from patients about the survey. Over several meetings, we reviewed survey items and discussed the ACP behavior change constructs and domains thought to be most important to retain. During these meetings, the team also flagged questions that appeared particularly difficult for study participants to understand.
Second, we determined the nonresponse or missing rate for each question as a marker of a difficult-to-answer question. Questions with nonitem response rates greater than 10% were flagged for possible deletion before factor analysis.
Third, we assessed for redundant items. Specifically, there appeared to be redundancy between the Action questions (e.g., “Have you already formally asked someone to be your medical decision maker,” with response options of “yes” or “no”) and the fifth response option in the five-point Likert scale for the corresponding Readiness questions (e.g., “How ready are you to formally ask someone to be your medical decision maker,” with “5” being “I already did it”). We compared responses to these potentially redundant items and decided to delete Action Measures with agreement greater than 80%. By removing the Action instead of the Readiness measures, we would retain the detail of the five-point Readiness measures and the ability to dichotomize responses as yes/no if desired. To further address redundancy, we then assessed correlations between the remaining survey items within and across ACP domains. Items that had correlation coefficients greater than 0.80 were removed before factor analysis.
Fourth, we looked for ceiling effects by assessing the proportion of patients who reported a “5” on the five-point Likert scale for any Process Measure. We used 70% with a response option of “5” to flag an item for removal.
Fifth, we used exploratory factor analysis using varimax rotation to identify factors in which the items naturally grouped. Factors with eigenvalues of more than one were retained.
These factors were reviewed, and then we selected items from each factor that either had the highest factor loading or met our face validity criteria for the most important ACP constructs determined a priori. Items are standardly removed from factor analysis if they have loading factors of less than 0.3. Factor analysis was conducted iteratively to create progressively shorter versions of the survey and ensure the appropriate items loaded on remaining factors.
In the sixth and final reduction analysis step, all items were systematically removed, one at a time, in separate analyses for each progressively shorter versions of the survey, to determine if removal of single items improved the Cronbach's alpha. We considered deleting items whose removal from the shortened surveys improved the Cronbach's alpha greater than 0.05.
Methods: Phase 2, Internal Consistency and Construct Validity
We first determined internal consistency (i.e., Cronbach's alpha) for all versions of the survey that resulted from Phase 1 analyses. Then, we measured construct validity in three ways. We assessed the correlations of the overall average five-point scores of the original, 82-item version of the survey with the progressively shorter versions using Pearson correlation coefficients (Fig. 1). The overall average five-point score was created using the five-point Likert scores for the Process Measures and by assigning a value of 5 to Action Measure response options of “yes” and a zero to a response option of “no.”
Next, we assessed whether each version of the survey was associated with patients' self-reports of prior planning, defined as having drafted a will, having made funeral plans, or completed an advance directive, using chi-square tests. We hypothesized that both the original and shorter versions of the survey would be associated with these pre-planning activities. Previous planning questions were asked before and separate from the survey.
Finally, we felt it was highly important that the shorter versions of the survey be able to detect change in response to an ACP intervention. The aforementioned ACP trials, from which Phase 2 recipient data were obtained, use the original 82-item version of the survey. As previously described, each study arm receives an intervention (i.e., controls receive an advance directive and the intervention arm receives a directive plus an ACP Web site).
Therefore, we expect an increase in survey scores regardless of study arm. To maintain blinding, we deleted the randomization identification and combined both arms into one cohort. We excluded individuals who had missing data for any survey item. We then assessed the change (delta) in total average five-point survey scores from baseline to one week after ACP intervention exposure and assessed the correlations of the deltas for the original, 82-item version with progressively shorter survey versions.
Statistical Analysis
For descriptive statistics, we used percentages, means, and standard deviations (SD). To determine item nonresponse and ceiling effects, we used percentages. For correlations, we used Pearson correlation coefficients with coefficients of 0.80 or more considered acceptable. For internal consistency, we used Cronbach's alpha with a coefficient of 0.80 or more considered acceptable. For factor analysis, we used varimax rotation with the PROC FACTOR procedures in SAS® and created scree plots with statements of “method = principal rotate = varimax scree.” For associations of the surveys with prior planning activities, we used chi-squared tests. To determine the ability of the surveys to detect change, we used Pearson correlation coefficients to assess the correlations of the deltas in average five-point survey scores. For Phase 2 only, we stratified our validation analyses on English and Spanish language.
Results
Participants
For Phase 1, 501 English-speaking patients from the US (n = 352) and Canada (n = 150) were included, and for Phase 2, 664 English and Spanish-speaking patients from the US were included (Table 2). Two hundred and four English-speaking participants from the US were included in both phases.
Face validity: The overall behavior change constructs prioritized by the team included more activating constructs, such as self-efficacy and readiness rather than knowledge and contemplation. The team also felt that readiness questions concerning being “ready to decide” were difficult for participants to understand, and two questions were flagged for poor understanding, being ready to decide about a decision maker (Table 1, Question 12) and flexibility for the decision maker (Question 65). In addition, several members felt that the ACP domains of flexibility for decision makers and asking doctors questions were less important than other standard aspects of ACP, and several members felt that flexibility was a new and difficult concept for many participants. Furthermore, the team unanimously felt that the Action Measures were redundant to the readiness questions and supported their deletion if corroborated by the analysis (as mentioned subsequently). Finally, the team felt that items about specific questions asked of a provider (i.e., risks, benefits, etc., Table 1, Questions 77–81) were not as helpful as asking about Readiness to ask questions in general (Table 1, Question 82). Although several items were flagged for possible deletion, none were deleted during the face validity stage (Fig. 1).
No items were deleted due to nonresponse because no item had more than 10% missing responses (mean nonresponse 3.1% [±1.2%], data not shown). Redundancy between Action items (yes/no) and corresponding Readiness items (five-point Likert scale with “5” indicating “I have already done it”) was high (mean 96.1%, ±SD 5.9%), and only one of 25 items had discrepancy more than 20% (Table 1, Question 80). Therefore, we decided to remove all 25 Action items, leaving 57 Process Measure items measured on a five-point Likert scale.
For the remaining 57 items, no Pearson correlation coefficients met the 0.80 threshold (data not shown). A correlation of 0.66 was found between items concerning readiness to choose flexibility for a decision maker and readiness to talk to the surrogate about flexibility (Table 1, questions 65 and 67). We removed Question 65 based on face validity and the correlation, leaving 56 items (Table 1, Fig. 1).
For the remaining 56 items, only one item concerning readiness to decide on a surrogate reached our threshold for a ceiling effect and was deleted (Table 1, Question 12, data not shown).
In factor analysis, the remaining 55 items loaded on 10 factors, which explained 68.8% of the variance (Appendices 1–5). Within these 10 factors, we chose 34 items that spanned all 10 factors, had factor loading values greater than 0.3 (all >0.45), and were prioritized based on behavior change constructs of readiness and self-efficacy in the domains of surrogates and quality of life per our face validity assessment (Appendix 1). In repeated factor analysis, the 34-item survey loaded on seven factors, which explained 66.7% of the variance. Using the earlier mentioned criteria, we then chose 15 items that spanned six factors. The 15-items loaded on three factors, which explained 61.7% of the variance. Using the earlier mentioned criteria, we then chose nine items within the three factors. The nine items loaded onto two factors that explained 59.9% of the variance. Finally, using the earlier mentioned criteria, we chose four items from two factors. These four items loaded onto one domain that explained 58.5% of the variance. All factor loading values in all analyses were greater than 0.3.
In the final item reduction step, no step-by-step deletion of any individual item from any of the progressively shorter versions of the survey resulted in an increase in the Cronbach's alpha more than 0.0002 (data not shown). Therefore, no survey versions were further item reduced.
Phase 2, Internal Consistency and Construct Validation
The Cronbach's alpha for the different versions of the survey overall, and by English and Spanish speakers, ranged from 0.85 to 0.97 (all P < 0.001, Table 3). Using the original, 82-item survey as the reference, the Pearson correlation coefficients of the overall mean five-point scores with shorter surveys versions were high (0.85–0.97, all P < 0.001, Table 4). Correlation coefficients also remained high for both English and Spanish speakers.
Table 3Overall Internal Consistency of the Original and Progressively Shorter Survey Versions in English and Spanish
The average five-point scores on all versions of the survey were higher for people who engaged in prior planning (all P < 0.001, Table 5). In addition, all versions were able to detect change in response to an ACP intervention. Correlations of the change scores for the original survey with progressively shorter versions were high overall and for both English and Spanish speakers (range 0.66–0.94, all P < 0.001) (Table 6).
Table 5Association of Survey Scores With Pre-Planning Activities
This is a blinded sample that combined data from three ongoing randomized trials and that includes both the control groups (i.e., given an advance directive only) and the intervention groups (i.e., given the PREPARE Web site plus an advance directive).
Mean change (deltas) obtained by calculating the difference between the average five-point scores at baseline to one-week after reviewing an ACP intervention. Based on an average five-point score. Process items scored on a five-point Likert scale and action items scored as “yes” = 5 and “no” = 0.
Mean change (deltas) obtained by calculating the difference between the average five-point scores at baseline to one-week after reviewing an ACP intervention. Based on an average five-point score. Process items scored on a five-point Likert scale and action items scored as “yes” = 5 and “no” = 0.
Mean change (deltas) obtained by calculating the difference between the average five-point scores at baseline to one-week after reviewing an ACP intervention. Based on an average five-point score. Process items scored on a five-point Likert scale and action items scored as “yes” = 5 and “no” = 0.
Mean (SD)
Correlation
P-Value
82 Items
0.4 (.7)
—
—
0.4 (.6)
—
—
0.4 (.7)
—
—
55 Items
0.3 (.6)
0.93
<0.001
0.3 (.6)
0.93
<0.001
0.3 (.7)
0.94
<0.001
34 Items
0.3 (.7)
0.89
<0.001
0.3 (.6)
0.88
<0.001
0.3 (.7)
0.90
<0.001
15 Items
0.3 (.7)
0.82
<0.001
0.3 (.7)
0.82
<0.001
0.3 (.8)
0.82
<0.001
Nine items
0.3 (.8)
0.74
<0.001
0.3 (.7)
0.75
<0.001
0.4 (.9)
0.75
<0.001
Four items
0.4 (1.0)
0.68
<0.001
0.4 (.9)
0.66
<0.001
0.5 (1.0)
0.70
<0.001
SD = standard deviation.
a This is a blinded sample that combined data from three ongoing randomized trials and that includes both the control groups (i.e., given an advance directive only) and the intervention groups (i.e., given the PREPARE Web site plus an advance directive).
b Mean change (deltas) obtained by calculating the difference between the average five-point scores at baseline to one-week after reviewing an ACP intervention. Based on an average five-point score. Process items scored on a five-point Likert scale and action items scored as “yes” = 5 and “no” = 0.
Using survey data from diverse study participants from the US and Canada, we used rigorous and systematic item reduction methods to create progressively shorter versions of the ACP Engagement Survey from the original 82-item version to a 55-, 34-, 15-, nine-, and four-item version. Then, using data from multicenter randomized trials among English- and Spanish-speaking patients in the US, we demonstrated that these progressively shorter versions of the survey retained high internal consistency and had high construct validity, including the ability to detect change in response to an ACP intervention.
The original development and validation of the 82-item ACP Engagement Survey allowed measurement of multiple ACP behaviors, in addition to advance directive completion. Although the original survey included a larger number of ACP behaviors and quantified behavior change constructs (i.e., knowledge, contemplation, self-efficacy, and readiness) for each individual ACP behavior, it was long with a mean administration time of 49 minutes.
The present study has helped to produce psychometrically sound shorter versions of the survey while still measuring a broad range of ACP behavior change constructs and domains. As the internal consistency and construct validity were slightly higher for longer versions of the survey, researchers may wish to use the longer versions (55, 34, and 15 items) to determine efficacy of a program or ACP tool or to detect nuanced differences in specific ACP behaviors. However, the shorter versions (i.e., nine and four items) remained psychometrically sound and may be appropriate to use for quality improvement initiatives in the clinical settings or research studies with limited resources. Decisions concerning which length of the survey to use will depend on the balance between the needed sample size to detect change in response to an ACP intervention, participant survey burden, and study resources.
The strengths of this study include the strong theoretical basis underpinning the original 82-item survey, which was informed by Social Cognitive and Behavior Change Theories. Additional strengths were the inclusion of a multidisciplinary team of ACP content experts and racially/ethnically diverse patients from multiple medical centers from two countries, the rigorous and systematic psychometric validation in two phases, and the ability of the shorter versions of the survey to detect change in response to an ACP intervention. Furthermore, we included English and Spanish speakers in Phase 2 construct validity assessments and validity remained robust in both languages.
This study has several limitations. Phase 1, item reduction, only included English speakers; however, construct validity remained high for Spanish speakers in Phase 2. In Phase 2, validation of the survey took place in only one area of the US, which may limit generalizability. Generalizability of our findings beyond North America is also unknown. Furthermore, in addition to rigorous psychometric analysis, our team of ACP experts used face validity to prioritize behavior change constructs and ACP domains. Other research groups may prioritize different ACP constructs and domains. Finally, we still do not know the scoring thresholds of the survey associated with patient-centered outcomes. Based on Social Cognitive and Behavior Change Theories, any increase in the average five-point score may be associated with clinically meaningful improvement in patient-centered outcomes, such as receipt of care that is consistent with patients' goals. However, further longitudinal studies will be needed to determine the exact thresholds for a full range of ACP behaviors that lead to improved ACP outcomes.
In conclusion, several shorter versions of the ACP Engagement Survey are valid, internally consistent, and able to detect change across a broad range of ACP behaviors and ACP domains. The surveys are also ready for use among English and Spanish speakers. Having several psychometrically sound shortened versions of the survey provides flexibility to researchers and quality improvement experts when choosing surveys to measure successful ACP programs and tools in the research and clinical setting.
Disclosures and Acknowledgments
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. Dr. Sudore, however, is supported in part by the following grants: NIH R01AG045043, PCORI-1306–01500, VA HSR&D 11–110-2, the American Cancer Society (ACS) #19659, and NIH U24NR014637. No funding bodies had a role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, and approval of the manuscript; or decision to submit the manuscript for publication. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs. The other authors declare no conflicts of interest.
Appendix.
Appendix 1Factor Analysis With the 55-Item Version of the Survey, n = 501
Domains
Subscale
Question #
Survey Question
Factor Loading
Retained for 34-Item Version
Factor 1
QOL
Self-efficacy
42
How confident are you that today you could talk with your medical decision maker about the care you would want if you were very sick or near the end of life?
0.7505
X
FLEX
Self-efficacy
62
How confident are you that today you could talk with your doctor about how much flexibility you want to give your medical decision maker?
0.72995
X
QOL
Self-efficacy
43
How confident are you that today you could talk with your doctor about the care you would want if you were very sick or near the end of life?
0.72002
X
FLEX
Self-efficacy
61
How confident are you that today you could talk with your decision maker about how much flexibility you want to give them?
0.70473
X
QOL
Self-efficacy
26
How confident are you that today you could talk with your doctor about whether or not certain health situations would make your life not worth living?
0.66252
X
QOL
Self-efficacy
25
How confident are you that today you could talk with your medical decision maker about whether or not certain health situations would make your life not worth living?
0.64755
X
DM
Self-efficacy
8
How confident are you that today you could ask someone to be your medical decision maker?
0.63294
X
DM
Self-efficacy
9
How confident are you that today you could talk with your doctor about who you want your medical decision maker to be?
0.63277
X
DM
Self-efficacy
10
How confident are you that today you could talk with your other family and friends about who you want your medical decision maker to be?
0.53885
X
Factor 2
QOL
Contemplation
38
How much have you thought about the care you would want if you were very sick or near the end of life?
0.73172
QOL
Contemplation
21
How much have you thought about whether or not certain health situations would make your life not worth living?
0.71662
QOL
Contemplation
39
How much have you thought about talking with your medical decision maker about the care you would want if you were very sick or near the end of life?
0.71441
QOL
Contemplation
22
How much have you thought about talking with your medical decision maker about whether or not certain health situations would make your life not worth living?
0.69707
QOL
Contemplation
24
How much have you thought about talking with your other family and friends about whether or not certain health situations would make your life not worth living?
0.57988
QOL
Readiness
29
How ready are you to decide whether or not certain health situations would make your life not worth living?
0.57048
X
QOL
Readiness
46
How ready are you to decide on the medical care you would want if you were very sick or near the end of life?
0.48871
X
Factor 3
QOL
Readiness
52
How ready are you to talk to your other family and friends about the kind of medical care you would want if you were very sick or near the end of life?
0.69998
X
QOL
Readiness
35
How ready are you to talk to your other family and friends about whether or not certain health situations would make your life not worth living?
0.6605
X
QOL
Self-efficacy
44
How confident are you that today you could talk with your other family and friends about the care you would want if you were very sick or near the end of life?
0.65418
X
FLEX
Self-efficacy
63
How confident are you that today you could talk with your other family and friends about how much flexibility you want to give your medical decision maker?
0.62537
X
FLEX
Readiness
71
How ready are you to talk to your other family and friends about how much flexibility you want to give your medical decision maker?
0.5855
QOL
Self-efficacy
27
How confident are you that today you could talk with your other family and friends about whether or not certain health situations would make your life not worth living?
0.58294
X
QOL
Contemplation
41
How much have you thought about talking with your other family and friends about the care you would want if you were very sick or near the end of life?
0.57117
DM
Readiness
18
How ready are you to talk to your other family and friends about who you want your medical decision maker to be?
0.55163
X
Factor 4
FLEX
Contemplation
58
How much have you thought about talking with your medical decision maker about how much flexibility you want to give them?
0.77612
FLEX
Contemplation
57
How much have you thought about the amount of flexibility you would want to give your medical decision maker?
0.73945
FLEX
Contemplation
59
How much have you thought about talking with your doctor about how much flexibility you want to give your decision maker?
0.69154
FLEX
Contemplation
60
How much have you thought about talking with other friends and family about how much flexibility you want to give your decision maker?
0.67834
Factor 5
QOL
Readiness
33
How ready are you to talk to your doctor about whether or not certain health situations would make your life not worth living?
0.78326
X
QOL
Readiness
50
How ready are you to talk to your doctor about the kind of medical care you would want if you were very sick or near the end of life?
0.76664
X
DM
Readiness
16
How ready are you to talk with your doctor about who you want your medical decision maker to be?
0.65307
X
FLEX
Readiness
69
How ready are you to talk to your doctor about how much flexibility you want to give your decision maker?
0.65113
X
QOL
Contemplation
23
How much have you thought about talking with your doctor about whether or not certain health situations would make your life not worth living?
0.56606
QOL
Contemplation
40
How much have you thought about talking with your doctors about the care you would want if you were very sick or near the end of life?
0.56388
Factor 6
DM
Knowledge
2
How well informed are you about what makes someone a good medical decision maker?
0.85653
X
DM
Knowledge
3
How well informed are you about the types of decisions that a medical decision maker may have to make for you in the future?
0.78301
X
DM
Knowledge
1
How well informed are you about who can be a medical decision maker?
0.7825
FLEX
Knowledge
55
How well informed are you about what it means to give a medical decision maker flexibility to make future decisions?
0.60476
FLEX
Knowledge
56
How well informed are you about the different amounts of flexibility a person can give their medical decision maker?
0.55696
Factor 7
DM
Contemplation
5
How much have you thought about asking someone to be your medical decision maker?
0.74107
DM
Contemplation
4
How much have you thought about who your medical decision maker should be?
0.69099
X
DM
Contemplation
7
How much have you thought about talking with your other family and friends about who you want your medical decision maker to be?
0.67899
DM
Contemplation
6
How much have you thought about talking with your doctors about who you want your medical decision maker to be?
0.59162
Factor 8
QOL
Readiness
54
How ready are you to sign official papers putting your wishes in writing about the kind of medical care you would want if you were very sick or near the end of life?
0.78719
X
QOL
Readiness
37
How ready are you to sign official papers putting your wishes in writing about whether or not certain health situations would make your life not worth living?
0.7453
FLEX
Readiness
73
How ready are you to sign official papers putting your wishes in writing about how much flexibility to give your decision maker?
0.70768
X
DM
Readiness
20
How ready are you to sign official papers naming a person or group of people to make medical decisions for you?
0.45441
X
Factor 9
QOL
Readiness
48
How ready are you to talk to your decision maker about the kind of medical care you would want if you were very sick or near the end of life?
0.60591
X
QOL
Readiness
31
How ready are you to talk to your decision maker about whether or not certain health situations would make your life not worth living?
0.59538
X
DM
Readiness
14
How ready are you to formally ask someone to be your medical decision maker?
0.54097
X
FLEX
Readiness
67
How ready are you to talk to your decision maker about how much flexibility you want to give them?
0.47456
X
FACTOR 10
QUEST
Contemplation
75
How much have you thought about questions you will ask your doctor to help make good medical decisions?
0.70726
QUEST
Knowledge
74
How well informed are you about the types of questions you can ask your doctor that will help you make a good medical decision?
0.67435
QUEST
Self-efficacy
76
How confident are you that today you could ask the right questions of your doctor to help make good medical decisions?
0.65978
X
QUEST
Readiness
82
How ready are you to ask your doctor questions to help you make a good medical decision?
0.64189
X
DM = medical decision maker; QOL = quality of life; FLEX = flexibility for the surrogate decision maker; QUEST = asking questions of medical providers.
Appendix 2Factor Analysis With the 34-Item Version of the Survey, n = 501
Domain
Subscale
Question #
Survey Question
Factor Loading
Retained for 15-Item Version
Factor 1
QOL
Self-efficacy
43
How confident are you that today you could talk with your doctor about the care you would want if you were very sick or near the end of life?
0.75753
X
FLEX
Self-efficacy
62
How confident are you that today you could talk with your doctor about how much flexibility you want to give your medical decision maker?
0.7489
X
QOL
Self-efficacy
26
How confident are you that today you could talk with your doctor about whether or not certain health situations would make your life not worth living?
0.71715
QOL
Self-efficacy
42
How confident are you that today you could talk with your medical decision maker about the care you would want if you were very sick or near the end of life?
0.67081
X
FLEX
Self-efficacy
61
How confident are you that today you could talk with your decision maker about how much flexibility you want to give them?
0.6225
X
QOL
Self-efficacy
25
How confident are you that today you could talk with your medical decision maker about whether or not certain health situations would make your life not worth living?
0.60905
DM
Self-efficacy
9
How confident are you that today you could talk with your doctor about who you want your medical decision maker to be?
0.54063
Factor 2
QOL
Readiness
29
How ready are you to decide whether or not certain health situations would make your life not worth living?
0.80307
QOL
Readiness
31
How ready are you to talk to your decision maker about whether or not certain health situations would make your life not worth living?
0.74796
QOL
Readiness
46
How ready are you to decide on the medical care you would want if you were very sick or near the end of life?
0.63459
QOL
Readiness
48
How ready are you to talk to your decision maker about the kind of medical care you would want if you were very sick or near the end of life?
0.61073
X
DM
Readiness
14
How ready are you to formally ask someone to be your medical decision maker?
0.45894
X
Factor 3
FLEX
Readiness
69
How ready are you to talk to your doctor about how much flexibility you want to give your decision maker?
0.76301
X
QOL
Readiness
33
How ready are you to talk to your doctor about whether or not certain health situations would make your life not worth living?
0.75602
QOL
Readiness
50
How ready are you to talk to your doctor about the kind of medical care you would want if you were very sick or near the end of life?
0.75397
X
DM
Readiness
16
How ready are you to talk with your doctor about who you want your medical decision maker to be?
0.7221
X
Factor 4
QOL
Self-efficacy
44
How confident are you that today you could talk with your other family and friends about the care you would want if you were very sick or near the end of life?
0.77791
FLEX
Self-efficacy
63
How confident are you that today you could talk with your other family and friends about how much flexibility you want to give your medical decision maker?
0.75515
QOL
Self-efficacy
27
How confident are you that today you could talk with your other family and friends about whether or not certain health situations would make your life not worth living?
0.66325
QOL
Readiness
52
How ready are you to talk to your other family and friends about the kind of medical care you would want if you were very sick or near the end of life?
0.64977
QOL
Readiness
35
How ready are you to talk to your other family and friends about whether or not certain health situations would make your life not worth living?
0.6048
DM
Readiness
18
How ready are you to talk to your other family and friends about who you want your medical decision maker to be?
0.49298
Factor 5
FLEX
Readiness
73
How ready are you to sign official papers putting your wishes in writing about how much flexibility to give your decision maker?
0.75295
QOL
Readiness
54
How ready are you to sign official papers putting your wishes in writing about the kind of medical care you would want if you were very sick or near the end of life?
0.66702
X
DM
Readiness
20
How ready are you to sign official papers naming a person or group of people to make medical decisions for you?
0.52597
X
FLEX
Readiness
67
How ready are you to talk to your decision maker about how much flexibility you want to give them?
0.48607
X
FLEX
Contemplation
58
How much have you thought about talking with your medical decision maker about how much flexibility you want to give them?
0.4728
Factor 6
DM
Self-efficacy
8
How confident are you that today you could ask someone to be your medical decision maker?
0.66856
X
DM
Self-efficacy
10
How confident are you that today you could talk with your other family and friends about who you want your medical decision maker to be?
0.59714
DM
Contemplation
4
How much have you thought about who your medical decision maker should be?
0.44845
Factor 7
DM
Knowledge
3
How well informed are you about the types of decisions that a medical decision maker may have to make for you in the future?
0.79318
DM
Knowledge
2
How well informed are you about what makes someone a good medical decision maker?
0.77068
QUEST
Self-efficacy
76
How confident are you that today you could ask the right questions of your doctor to help make good medical decisions?
0.6021
X
QUEST
Readiness
82
How ready are you to ask your doctor questions to help you make a good medical decision?
0.39666
X
DM = medical decision maker; QOL = quality of life; FLEX = flexibility for the surrogate decision maker; QUEST = asking questions of medical providers.
Appendix 3Factor Analysis With the 15-Item Version of the Survey, n = 501
Domain
Subscale
Question #
Survey Question
Factor Loading
Retained for Nine-Item Version
Factor 1
DM
Readiness
14
How ready are you to formally ask someone to be your medical decision maker?
0.78782
X
DM
Readiness
20
How ready are you to sign official papers naming a person or group of people to make medical decisions for you?
0.77456
X
QOL
Readiness
48
How ready are you to talk to your decision maker about the kind of medical care you would want if you were very sick or near the end of life?
0.69771
X
QOL
Readiness
54
How ready are you to sign official papers putting your wishes in writing about the kind of medical care you would want if you were very sick or near the end of life?
0.58319
X
DM
Self-efficacy
8
How confident are you that today you could ask someone to be your medical decision maker?
0.58285
X
Factor 2
FLEX
Self-efficacy
62
How confident are you that today you could talk with your doctor about how much flexibility you want to give your medical decision maker?
0.81753
QOL
Self-efficacy
43
How confident are you that today you could talk with your doctor about the care you would want if you were very sick or near the end of life?
0.7937
X
QOL
Self-efficacy
42
How confident are you that today you could talk with your medical decision maker about the care you would want if you were very sick or near the end of life?
0.65406
X
FLEX
Self-efficacy
61
How confident are you that today you could talk with your decision maker about how much flexibility you want to give them?
0.65401
QUEST
Self-efficacy
76
How confident are you that today you could ask the right questions of your doctor to help make good medical decisions?
0.63063
Factor 3
FLEX
Readiness
69
How ready are you to talk to your doctor about how much flexibility you want to give your decision maker?
0.81969
QOL
Readiness
50
How ready are you to talk to your doctor about the kind of medical care you would want if you were very sick or near the end of life?
0.75862
X
DM
Readiness
16
How ready are you to talk with your doctor about who you want your medical decision maker to be?
0.71665
X
FLEX
Readiness
67
How ready are you to talk to your decision maker about how much flexibility you want to give them?
0.57136
QUEST
Readiness
82
How ready are you to ask your doctor questions to help you make a good medical decision?
0.43226
DM = medical decision maker; QOL = quality of life; FLEX = flexibility for the surrogate decision maker; QUEST = asking questions of medical providers.
Appendix 4Factor Analysis With the Nine-Item Version of the Survey, n = 501
Domain
Subscale
Question #
Survey Question
Factor Loading
Retained for Four-Item Version
Factor 1
QOL
Readiness
50
How ready are you to talk to your doctor about the kind of medical care you would want if you were very sick or near the end of life?
0.78746
X
DM
Readiness
16
How ready are you to talk with your doctor about who you want your medical decision maker to be?
0.72991
QOL
Readiness
48
How ready are you to talk to your decision maker about the kind of medical care you would want if you were very sick or near the end of life?
0.67932
X
QOL
Readiness
54
How ready are you to sign official papers putting your wishes in writing about the kind of medical care you would want if you were very sick or near the end of life?
0.66582
X
DM
Readiness
20
How ready are you to sign official papers naming a person or group of people to make medical decisions for you?
0.58578
X
DM
Readiness
14
How ready are you to formally ask someone to be your medical decision maker?
0.58201
Factor 2
QOL
Self-efficacy
42
How confident are you that today you could talk with your medical decision maker about the care you would want if you were very sick or near the end of life?
0.86799
DM
Self-efficacy
8
How confident are you that today you could ask someone to be your medical decision maker?
0.81362
QOL
Self-efficacy
43
How confident are you that today you could talk with your doctor about the care you would want if you were very sick or near the end of life?
0.67436
DM = medical decision maker; QOL = quality of life.
Appendix 5Factor Analysis With the Four-Item Version of the Survey, n = 501
Domain
Subscale
Question #
Survey Question
Factor Loading
Factor 1
QOL
Readiness
48
How ready are you to talk to your decision maker about the kind of medical care you would want if you were very sick or near the end of life?
0.79526
QOL
Readiness
50
How ready are you to talk to your doctor about the kind of medical care you would want if you were very sick or near the end of life?
0.68681
QOL
Readiness
54
How ready are you to sign official papers putting your wishes in writing about the kind of medical care you would want if you were very sick or near the end of life?
0.79998
DM
Readiness
20
How ready are you to sign official papers naming a person or group of people to make medical decisions for you?
0.78022
DM = medical decision maker; QOL = quality of life.
A comparative, retrospective, observational study of the prevalence, availability, and specificity of advance care plans in a county that implemented an advance care planning microsystem.
Promoting advance care planning as health behavior change: development of scales to assess decisional balance, medical and religious beliefs, and processes of change.
31209-X&id=gr1.jpg){kind=link}