Advertisement

Validation of a Measure of Family Experience of Patients with Serious Illness: The QUAL-E (Fam)

  • Karen E. Steinhauser
    Correspondence
    Address correspondence to: Karen E. Steinhauser, PhD, Center for Palliative Care, VA and Duke Medical Centers, 2424 Erwin Road, Suite 1105, Box 2720, Durham, NC 27705, USA.
    Affiliations
    Center for Health Services Research in Primary Care, Durham Veterans Administration Medical Center, Duke University, Durham, North Carolina, USA

    Division of General Internal Medicine, Duke University, Durham, North Carolina, USA

    Division of Palliative Care, Duke University, Durham, North Carolina, USA

    Center for the Study of Aging and Human Development, Duke University, Durham, North Carolina, USA
    Search for articles by this author
  • Corrine I. Voils
    Affiliations
    Center for Health Services Research in Primary Care, Durham Veterans Administration Medical Center, Duke University, Durham, North Carolina, USA
    Search for articles by this author
  • Hayden B. Bosworth
    Affiliations
    Center for Health Services Research in Primary Care, Durham Veterans Administration Medical Center, Duke University, Durham, North Carolina, USA

    Division of General Internal Medicine, Duke University, Durham, North Carolina, USA
    Search for articles by this author
  • James A. Tulsky
    Affiliations
    Center for Health Services Research in Primary Care, Durham Veterans Administration Medical Center, Duke University, Durham, North Carolina, USA

    Division of General Internal Medicine, Duke University, Durham, North Carolina, USA

    Division of Palliative Care, Duke University, Durham, North Carolina, USA

    Center for the Study of Aging and Human Development, Duke University, Durham, North Carolina, USA

    School of Nursing, Duke University, Durham, North Carolina, USA
    Search for articles by this author
Open AccessPublished:May 22, 2014DOI:https://doi.org/10.1016/j.jpainsymman.2014.04.006

      Abstract

      Context

      Family members of seriously ill patients experience significant burden as they advocate with providers and participate in key decisions for loved ones. Most assessments focus on patient experience, yet family members' own quality of experience is central to comprehensive care.

      Objectives

      This study validated a measure of the quality of family experience, the QUAL-E (Fam), in palliative care.

      Methods

      We enrolled family members of terminally ill patients admitted to general medicine services at two hospitals. Items were based on foundational work originally done for the patient instrument, QUAL-E, and interviews of family members of terminally ill patients. Cognitive interviewing was used to refine items, which then underwent formal testing factor analysis. In the first sample, we assessed factor structure. A subsequent sample established predictive validity and test-retest reliability.

      Results

      The initial item pool was reduced to a four-domain, 35-item scale and administered to the validation sample. Further analyses produced a final brief scale comprising 17 items, demonstrating appropriate convergent and divergent validity. Test-retest reliability demonstrated expected levels of stability in a highly changeable population. The scale provides an assessment of family experience and includes subscales assessing relationship with health care provider and completion. Additional scale items assess symptom experience and issues of preparation.

      Conclusion

      The QUAL-E (Fam) is a companion instrument to the patient QUAL-E measure of quality of life at the end of life and is part of a package of assessment tools that can help evaluate the entire patient experience and contribute to quality care.

      Key Words

      Introduction

      Patients with serious illness do not suffer in isolation; they are cared by family members and loved ones who take on huge responsibilities and bear stress under this burden. Most measurement efforts focus on assessing the quality of patient experience.
      • Aaronson N.K.
      Quality of life assessment in clinical trials: methodologic issues.
      • Cella D.F.
      • Tulsky D.S.
      • Gray G.
      • et al.
      The Functional Assessment of Cancer Therapy scale: development and validation of the general measure.
      • Cohen S.
      • Mount B.
      Quality of life assessment in terminal illness: defining and measuring subjective well-being in the dying.
      • Cohen S.
      • Mount B.
      • Strobel M.G.
      • et al.
      The McGill Quality of Life Questionnaire: a measure of quality of life for people with advanced disease. A preliminary study of validity and acceptability.
      • Ferrans C.
      • Powers M.
      Quality of Life Index: development and psychometric properties.
      • Keall R.M.
      • Butow P.N.
      • Steinhauser K.E.
      • Clayton J.M.
      Discussing life story, forgiveness, heritage, and legacy with patients with life-limiting illnesses.
      • Steinhauser K.E.
      Measuring end-of-life care outcomes prospectively.
      • Steinhauser K.E.
      • Bosworth H.B.
      • Clipp E.C.
      • et al.
      Initial assessment of a new instrument to measure quality of life at the end of life.
      Patient accounts may be the gold standard for assessing patient quality of life through recording their subjective experience prospectively, in real time. To assess quality of care, instruments have relied on surrogate accounts, usually from family members, either as a proxy in real time or retrospectively, during periods of bereavement.
      • Casarett D.
      • Pickard A.
      • Bailey F.A.
      • et al.
      A nationwide VA palliative care quality measure: the family assessment of treatment at the end of life.
      • Teno J.M.
      • Clarridge B.
      • Casey V.
      • Edgman-Levitan S.
      • Fowler J.
      Validation of Toolkit After-Death Bereaved Family Member Interview.
      • Connor S.R.
      • Teno J.
      • Spence C.
      • Smith N.
      Family evaluation of hospice care: results from voluntary submission of data via website.
      • Portenoy J.
      • Teno J.M.
      Hispanic language version of the family evaluation of hospice care.
      • Rhodes R.L.
      • Mitchell S.L.
      • Miller S.C.
      • Connor S.R.
      • Teno J.M.
      Bereaved family members' evaluation of hospice care: what factors influence overall satisfaction with services?.
      • Teno J.M.
      Measuring end-of-life care outcomes retrospectively.
      However, consistent with the philosophy of palliative care, family members' experience ought to be captured—not only as proxies but also as care recipients.
      • Steinhauser K.E.
      • Clipp E.C.
      • Tulsky J.A.
      Evolution in measuring the quality of dying.
      Family members often become intermediaries with health care teams, advocates, and key decision makers on behalf of their loved ones.
      In prior work, we used data gathered in qualitative interviews to develop an instrument to assess family experience.
      • Bosworth H.B.
      • Steinhauser K.E.
      • Orr M.
      • et al.
      Congestive heart failure patients’ perceptions of quality of life: the integration of physical and psychosocial factors.
      • Steinhauser K.E.
      • Clipp E.C.
      • McNeilly M.
      • et al.
      In search of a good death: observations of patients, families, and providers.
      In this article, we present the validation of that measure including cognitive interviewing results to refine the initial draft and subsequent assessments of factor structure and convergent and divergent validity.

      Methods

      Design

      Previous qualitative work led to a draft QUAL-E (Fam) measure of 51 items. In this study, we used cognitive interviewing
      • DeMaio T.
      • Rothgeb J.
      Cognitive interviewing techniques: In the lab and in the field.
      to refine instrument item structure and increase scale reliability. Subsequently, we obtained initial evidence of reliability and validity using factor analysis, test-retest reliability, and correlations with other measures. This evidence was gathered from two consecutive samples. In the first sample, we assessed factor structure. A subsequent sample established predictive validity and test-retest reliability.

      Setting and Participants

      Family members of terminally ill patients were recruited from the medical services of two sites: the Durham Veterans Administration (VA) and Duke University Medical Centers. Eligibility criteria included cognitively intact loved ones (score of 7 or higher on the Short Portable Mental Status Questionnaire)
      • Pfieffer E.
      A short portable mental status questionnaire for the assessment of organic brain deficit in elderly patients.
      of patients with advanced cancer, congestive heart failure, chronic obstructive pulmonary disease, or other life-limiting illness, and for whom the provider would not be surprised if the patient died within six months. Subjects were reimbursed $20 for their participation.

      Cognitive Interview Analyses

      A trained interviewer conducted one-on-one interviews with participants in an iterative fashion. The draft instrument was administered to five subjects, revised, and then readministered to another five subjects, iteratively, until no additional changes were necessary. In later rounds, fewer subjects were included as fewer substantive changes were being made.
      After each interview, response summaries and problematic issues as well as corresponding exemplars were flagged for discussion with the investigative team. These issues included awkward or ambiguous question wording, lack of conceptual clarity, inadequate or unnecessary response categories, and the need for more questions. The investigative team reviewed these data and made necessary changes to item wording, or, when necessary, item reduction.
      • DeMaio T.
      • Rothgeb J.
      Cognitive interviewing techniques: In the lab and in the field.
      We conducted 32 cognitive interviews with 24 VA and eight Duke participants. Five caregivers were males and 27 females. Forty-four percent of respondents were spouses, 28% adult children, 3% parents, 6% friend/caretaker, and 19% unknown. Fifty percent of respondents' loved ones were ill with cancer, 9% with end-stage renal disease, 14% with end-stage liver disease, 2% with unrecorded diagnoses, and 6% with congestive heart failure.
      After identifying items with consistent readability and interpretability, we retained an initial 35-item QUAL-E (Fam) instrument representing four domains: symptom experience (five items), relationship with health care provider (HCP) (20 items), completion (three items), and preparation (five items), with two additional items assessing how the respondent would rate the experience with the health care team and how the respondent was doing overall (Table 1). The relationship with HCP subscale was particularly lengthy because we were testing whether items using the terms “doctor” vs. “nurse” would be evaluated differently.
      Table 1Items Submitted for Factor Analyses for Validation Phases 1 and 2
      Phase 1 (35 Items)ItemPhase 2 (25 Items)Item
      Symptom experienceDuring the last week how often has Patient experienced (smyp)?sympexpDuring the last week how often has Patient experienced (smyp)?
      Symptom severityDuring the last week, on average, how severe has (smyp) been?sympsevDuring the last week, on average, how severe has (smyp) been?
      Symptom importanceDuring the past week, how much has (symp) interfered with hi/her life?sympintDuring the past week, how much has (symp) interfered with his/her life?
      Symptom in futureHow concerned are you about the Patient's (symp) in the future?
      Symptom totalIn your opinion, how much suffering is Patient experiencing?symptotIn your opinion, how much suffering is Patient experiencing?
      Answer questionsDuring the last week, how often has the health care team been available to answer your questions about Patient?
      Expect about illnessHow much have you been told about what to expect as Patient's illness progress?expectillHow much have you been told about what to expect as Patient's illness progress?
      Involved in decisionHow often does the health care team involve you in making decisions about Patient's care?involdecHow often does the health care team involve you in making decisions about Patient's care?
      Informed about conditionHow often does the health care team keep you informed about Patient's condition?informedHow often does the health care team keep you informed about Patient's condition?
      Confused about careHow often are you confused about what is happening with Patient's care?confusedHow often are you confused about what is happening with Patient's care?
      Communicate in difficult languageHow often does the health care team communicate in language that is difficult to understand?communicateHow often does the health care team communicate in language that is difficult to understand?
      Conflict about informationHow often does the health care team give conflicting information about the Patient's care?conflict infoHow often does the health care team give conflicting information about the Patient's care?
      Family agreeHow much does your family agree with decisions about Patient's care?
      MD whole personHow often do the doctors treat Patient as a whole person rather than a disease?hcteamwholeHow often does the health care team treat Patient as a whole person rather than a disease?
      RN whole personHow often do the nurses treat Patient as a whole person rather than a disease?
      MD dignityHow often do the doctors treat Patient in a way that maintains his/her dignity?hcteamdigitHow often does the health care team treat Patient in a way that maintains his/her dignity?
      RN dignityHow often do the nurses treat Patient in a way that maintains his/her dignity?
      MD do likeHow often do the doctors do everything you would like in Patient's carehcteamdolikeHow often does the health care team do everything you would like in Patient's care
      RN do likeHow often do the nurses do everything you would like in Patient's care
      PrivacyHow often do you have the privacy you need with Patient?
      ConcernHow often does the health care team respond to your concerns about Patient?concernHow often does the health care team respond to your concerns about Patient?
      Ask how doing?How often does the health care team ask how you are doing?askyoudoHow often does the health care team ask how you are doing?
      MD supportHow much emotional support do the doctors give you?hcteamsupportHow such emotional support does the health care team give you?
      RN supportHow much emotional support do the nurses give you?
      History of illnessHow often does the health care team know enough about the history of Patient's illness to provide the best care possible?
      Positive differenceHow much of the time are you able to make a positive difference for Patient?posdiffHow much of the time are you able to make a positive difference for Patient?
      Meaningful interactionHow much of the time are you able to have meaningful interaction with Patient?interactHow much of the time are you able to have meaningful interaction with Patient?
      Say important thingsHow much of the time are you able to say important things to Patient?sayimpHow much of the time are you able to say important things to Patient?
      Family supportHow much do you feel supported by family?
      Financial strainHow much do you feel financial strain related to Patient's illness?finstrainHow much do you feel financial strain related to Patient's illness?
      Prepared for futureHow prepared are you emotionally for the future, no matter what happens?preparedHow prepared are you emotionally for the future, no matter what happens?
      PeaceHow much do you feel at peace?peaceHow much do you feel at peace?
      Peace about careHow much do you feel at peace with Patient's care?peacecareHow much do you feel at peace with Patient's care?
      Overall experienceOverall, how would you rate your experience with the health care team?overallexpOverall, how would you rate your experience with the health care team?
      How are you overallGiven the circumstances, how are you doing overall?youoverallGiven the circumstances, how are you doing overall?
      MD = medical doctor; RN = registered nurse.

      Instrument Validation

      Instrument validation occurred in two consecutive samples. After the first sample, data were analyzed to inform item reduction and identify basic factor structure. In the second sample, the revised QUAL-E (Fam) was administered with a small battery of comparison measures to provide evidence of reliability and validity.

      Physical Symptoms

      The M. D. Anderson Symptom Inventory (MDASI) is a 19-item scale that measures symptom prevalence as well as severity and degree of interference. It assesses 13 symptoms commonly experienced by seriously ill patients, such as pain, fatigue, nausea, dyspnea, and so on. Internal consistency was high.
      • Armstrong T.S.
      • Mendoza T.
      • Gning I.
      • et al.
      Validation of the M.D. Anderson Symptom Inventory Brain Tumor Module (MDASI-BT).

      Relationship with HCP

      We used four scales (24 items) from the Primary Care Assessment Survey (PCAS) to measure relationship quality.
      • Safran D.G.
      • Kosinski M.
      • Tarlov A.R.
      • et al.
      The Primary Care Assessment Survey: tests of data quality and measurement performance.
      • Safran D.G.
      • Taira D.A.
      • Rogers W.H.
      • et al.
      Linking primary care performance to outcomes of care.
      These scales—communication, interpersonal treatment, physician's knowledge of the patient, and patient trust—correlate well with outcomes of care such as adherence and satisfaction. With preliminary testing revealing a strong quality of care focus, we chose to use a quality of care assessment tool for comparison.

      Spiritual Well-Being

      We used the Functional Assessment of Chronic Illness Therapy—Spiritual Well-Being (FACIT-Sp) scale, a 12-item measure of faith and meaning and purpose as a comparison with the QUAL-E (Fam) completion subscale. The FACIT-Sp subscales correlate with the QUAL-E completion subscale.
      • Peterman A.H.
      • Fitchett G.
      • Brady M.J.
      • Hernandez L.
      • Cella D.
      Measuring spiritual well-being in people with cancer: the functional assessment of chronic illness therapy–Spiritual Well-being Scale (FACIT-Sp).
      Internal consistency is high.
      In the second sample, the instrument was readministered by phone one week later, to a participant subset, to assess test-retest reliability. In addition, we collected demographic and clinical data, including age, gender, ethnicity, education, income and religious affiliation, and patient's primary diagnosis, lengths of stay, relationship of the family member to the patient, and level of communication with the patient.

      Instrument Analysis

      Response Distributions

      For all items, we examined mean, median, range, skewness, and kurtosis. From Phase 1 to Phase 2, we reduced the original pool of 35 items to 25 by removing three highly skewed and seven redundant items.

      Factor Structure

      The item pool comprised individual stand-alone items as well as effect and causal indicators to measure latent variables. Indicators are used because latent variables (e.g., depression) cannot be assessed directly. Effect indicators are scale items that reflect one's standing on the latent variable
      • Bollen K.
      • Lennox R.
      Conventional wisdom on measurement: a structural equation perspective.
      and comprise a series of similarly worded items to tap into the underlying latent construct. Thus, they are positively correlated with one another, an important requirement of internal consistency reliability and factor analysis. In contrast, causal indicators are items that represent different aspects of an underlying latent construct but are insufficient individually to fully describe that latent construct. Because causal indicators measure distinct facets of a latent construct, they may be negatively or positively correlated or not correlated at all. Consequently, internal consistency and factor analysis are inappropriate. Instead, test-retest reliability is used to establish reliability. Although most latent variables are assessed by effect indicators, many latent constructs, including quality of life, stress, and adherence, are measured appropriately by a mix of causal and effect indicators.
      • Fayers P.M.
      • Hand D.J.
      • Bjordal K.
      • Groenvold M.
      Causal indicators in quality of life research.
      • Voils C.I.
      • Hoyle R.H.
      • Thorpe C.T.
      • Maciejewski M.L.
      • Yancy Jr., W.S.
      Improving the measurement of self-reported medication nonadherence.
      In scales comprising both effect and causal indicators, the effect indicators are combined into subscale scores and submitted to factor analysis and internal consistency, whereas causal indicators are evaluated individually or as a composite score. Two items were designed as stand-alone items: overall experience and you overall.
      Because both one's relationship with health care professionals and sense of completion were measured by effect indicators, they were evaluated as subscales. This determination was made based on previous qualitative work on construct validity and verified by large positive interitem correlations. In contrast, those items assessing symptoms and preparation were conceptually distinct, which was verified through small interitem correlations. For example, the four symptom items gave rise to the latent construct of overall symptom experience and were not highly positively correlated (ranging from 0.13 to 0.35). The four preparation items also were weakly correlated (ranging from 0.08 to 0.38). Of note, we originally expected to retain more preparation items to adequately represent the construct based on our previous qualitative work. In our qualitative data from bereaved family members, preparation included items such as knowing what to expect about the physical process of dying and the emotional process of dying as well as being financially prepared.
      • Bosworth H.B.
      • Steinhauser K.E.
      • Orr M.
      • et al.
      Congestive heart failure patients’ perceptions of quality of life: the integration of physical and psychosocial factors.
      As we moved from bereaved family members' retrospective accounts of what was important to developing a prospective measure of quality of family experience before death, we constructed items focused on the present understanding of illness course, without reference to death. The items assessing information and knowing what to expect were dependent on what had been communicated to family members via the health care team and, as a result, were correlated with other items in the relationship with HCP subscale, rather than preparation items as originally intended. The remaining preparation issues were independent of the health care team and were intrinsic to the family member (e.g., financial strain, sense of peace). Because these issues determine preparation rather than vice versa, they were deemed causal indicators.
      Because the foundational qualitative work suggested a priori domains contributing to quality of family experience, we used confirmatory rather than exploratory factor analysis in both study phases, which allows hypotheses about the factor structure to be directly tested and either confirmed or refuted. We assessed model fit using several criteria: Chi-square; Bentler's comparative fit index (CFI), a robust measure based on the concentrality of the Chi-square index; and the goodness-of-fit index (GFI), a maximum likelihood estimation of the relative amount of variance and covariance accounted for by the model.
      • Bentler P.
      Comparative fit indices in structural models.
      • Bentler P.
      • Bonett D.
      Significance tests and goodness-of-fit in the analysis of covariance structures.
      • Joreskog K.
      • Sorbom D.
      • Lisrel V.
      Analysis of linear structural relationships by the method of maximum likelihood.
      CFI and GFI values range from 0 to 1.0, with values closest to 1.0 suggesting a better fit. Finally, the root mean square approximation of error was examined as a measure of model discrepancy per degrees of freedom.

      Steiger J, Lind J. Statistically based tests for the number of common factors. Paper presented at the annual meeting of the Psychometric Society, Iowa City, IA, 1980.

      Values from 0.06 to 0.08 indicate a reasonable fit; those at 0.05 or below indicate a good fit.
      • Browne M.
      • Cudeck R.
      Alternative ways of assessing model fit.
      Once the factor structure was confirmed, we calculated the internal consistency—reliability (Cronbach's alpha) of the subscales. We also examined test-retest reliability of the subscales and the individual causal indicators (i.e., symptom items).
      We then examined correlations among QUAL-E (Fam) subscales, hypothesizing no relationships among unrelated subscales (i.e., symptoms and completion) and weak to moderate correlations (r = 0.01–0.03) among related scales or those with conceptual overlap (i.e., completion and preparation). Finally, we examined convergent and divergent validity of QUAL-E (Fam) subscales with additional measures administered. We expected that symptom items would be positively associated with the MDASI items, HCP subscales should be correlated with items in the PCAS, and peace items and the completion subscales were compared with FACIT-Sp measure items.

      Results

      Phase 1 Validation

      One hundred eighty-four caregivers were approached for participation in the initial validation of the QUAL-E (Fam). Twenty-two were unable to be contacted, 14 refused, three were cognitively unable, and 145 consented. Seventy-four caregivers were enrolled from the Durham VA Medical Center and 71 from the Duke University Medical Center; 113 (78%) were women, 37 (26%) African American, and 91 (63%) were Caucasian respondents. Sixty-nine (48%) participants reported a high school education or less, 26 (18%) had some college, 26 (18%) a college degree, and 21 (15%) a graduate or professional degree (Table 2).
      Table 2Instrument Validation Samples
      CharacteristicsPhase 1Phase 2
      n = 141 (%)n = 244 (%)
      Gender
       Male30 (21.0)42 (16.8)
       Female113 (79.0)208 (83.2)
      Ethnicity
       Caucasian91 (64.5)142 (58.4)
       African American37 (26.2)77 (31.7)
       Other13 (9.3)24 (9.9)
      Marital status
       Married/living with partner111 (79.3)194 (79.2)
       Widowed5 (3.6)8 (3.2)
       Divorced/separated15 (10.7)25 (10.2)
       Never married9 (6.4)18 (7.4)
      Education
       Less than HS/HS or GED68 (48.6)105 (43.0)
       AA26 (18.3)63 (25.8)
       BA/BS26 (18.3)47 (19.3)
       Graduate/professional21 (14.8)29 (11.9)
      Income
       $0–99994 (3.1)18 (8.4)
       $10,000–19,99920 (15.9)32 (14.9)
       $20,000–34,99931 (24.6)49 (22.9)
       $35,000–49,99917 (13.5)34 (15.9)
       More than $50,00054 (42.9)81 (37.9)
      Religion
       Protestant120 (85.7)186 (77.5)
       Jewish1 (0.7)2 (0.8)
       Roman Catholic6 (4.3)18 (7.5)
       Other4 (2.9)22 (9.2)
       None9 (6.4)12 (5.0)
      Mood
       Not at all depressed82 (58.2)150 (61.5)
       Slightly depressed37 (26.2)55 (22.5)
       Moderately depressed17 (12.1)27 (11.1)
       Quite a bit depressed5 (3.5)11 (4.5)
       Extremely depressed0 (0)1 (0.4)
      HS = High School; GED = General Educational Development; AA = Associate of Arts; BA = Bachelor of Arts; BS = Bachelor of Science.
      Note: Items may not equal 100 because of rounding.

      Item Distributions and Reductions

      After cognitive interviewing, the instrument draft included 35 items. That initial version was submitted to family members for the first round of validation and factor analyses. Reduction from the initial 35 items to 25 that were subsequently administered in Phase 2 resulted from two issues. First, in the 35-item version, items assessing relationship with HCP were represented by two versions, one asking family members about their relationship with nurses and one asking about their relationship with doctors. We wanted to test whether family members' responses differed by the HCP's specific role. This was not the case as most of these items were highly correlated (i.e., ranging from 0.47 to 0.70, P < 0.0001). One exception was a question about being treated with dignity, for which the nurse and doctor versions were significantly but weakly correlated (0.29, P < 0.0005). Cognitive interviews with members of the target population also pointed to redundancy between doctor and nurse versions. Therefore, we included only one version of each question that referred to the HCP to capture both doctors and nurses. This resulted in the loss of three items. Another seven items were highly skewed: sympfut, anwquest, communicate, famagree, privacy, support, ptsillness, and famsupp (Table 1). Removal of these 10 items resulted in a 25-item version that was examined in a second round of validation.

      Domains

      Twenty-three of the original 35 items were effect indicators and thus were entered into factor analysis. Factor analyses resulted in reduced items and revealed two distinct domains (Table 3) comprising 15 items.
      Table 3Confirmatory Factor Analysis Loadings
      ItemPhase 1Phase 2
      Factor 1: Relationship with HCP
       Being told what to expect about illness0.740.74
       Being involved in decision making0.680.86
       Being kept informed about loved one's care0.840.85
       Health care team responds to concerns0.620.57
      Factor 2: Completion
       Being able to make a positive difference for patients0.490.39
       Able to have meaningful interaction0.970.94
       Being able to say important things to patient0.560.60
      HCP = health care provider.
      Round 1: X2(13) = 21.8, P = 0.06; comparative fit index = 0.97; root mean square approximation of error = 0.07 (90% CI = 0.00, 0.12).
      Round 2: X2(13) = 9.6, P = 0.73; comparative fit index = 1.0; root mean square approximation of error = 0.00 (90% CI = 0.00, 0.05).
      The first factor (12 items) measured relationship with HCP and included 1) being told what to expect about illness, 2) being involved in decision making, 3) being kept informed about loved one's care, 4) confusion about what's happening in patient care, 5) the team communicating in language that is difficult to understand, 6) getting conflicting information, 7) the patient being treated as a whole person, 8) the patient being treated with dignity, 9) health care team does everything caregiver would like in patient care, 10) health care team responds to concerns, 11) being asked about how you as a family member are doing, and 12) health care team gives emotional support.
      The second factor (three items) assessed completion. It included questions rating 1) being able to make a positive difference for patients, 2) able to have a meaningful interaction, and 3) being able to say important things to the patient.
      The resulting version of the QUAL-E (Fam) included 25 items: eight causal indicators for symptoms and preparation (assessing financial strain, level of preparedness, level of peace, and level of peace about care), 15 effect indicators, and two individual items representing quality of experience (overall experience) and quality of life (you overall).

      Phase 2 Validation

      Of 372 referred subjects, 79 could not be contacted, 38 refused, five were ineligible because of cognitive impairment, and 250 were enrolled (85%). One hundred twenty-seven (51%) were from the Durham VA and 123 from Duke University Medical Centers; 208 (83%) were women, 77 (31%) African American, and 142 (57%) Caucasian. One hundred five (42%) participants reported a high school education or less, 63 (25%) had some college, 47 (19%) a college degree, and 29 (12%) a graduate or professional degree (Table 2).

      Factor Structure

      GFI, CFI, and root mean square approximation of error measures show excellent model fit (Table 3). Consistent with Phase 1, factor analysis of the 15 effect indicators revealed two distinct factors (Table 3) and produced a further reduced item list. Seven relationship items did not improve the fit of the construct and were removed to improve parsimony and respondent burden. Thus, four items (α = 0.84) measured relationship with HCP. Three items (α = 0.65) assessed completion.

      Test-Retest Reliability

      The test-retest reliability of the relationship with HCP subscale was 0.66 and of the completion subscale was 0.51 (Table 4). Individually retained items in the preparation domain included financial strain (r = 0.61), prepared (r = 0.63), peace (r = 0.53), and peace with care (r = 0.55). We confirmed previous research in this population revealing that symptoms in this rapidly changing population are highly variable and do not warrant test-retest assessment.
      • Steinhauser K.
      • Clipp E.
      • Bosworth H.
      • et al.
      Measuring quality of life at the end of life: validation of the QUAL-E.
      Test-retest reliabilities were 0.58 for the item assessing the quality of their overall experience and 0.41 for the item rating how they were doing overall.
      Table 4Test-Retest Reliabilities (n = 100)
      Scale Componentr
      Subscales
       Completion0.51
       Relationship with HCP0.66
      Individual items
       Financial strain0.61
       Prepared0.63
       Peace0.54
       Peace care0.55
      Overall experience0.58
      Doing overall0.41
      Percentages changing in the last week
       Medication change44
       Treatment change27
       New hospitalization2
      HCP = health care provider.

      Convergent and Discriminant Validity

      We compared all causal indicators and the two QUAL-E (Fam) subscales with responses to four subscales in the PCAS: trust, communication, comprehensive knowledge, and interpersonal (Table 5). Relationship with HCP was significantly related to all PCAS subscales, with r's ranging between 0.54 and 0.69, showing strong convergent validity, as expected.
      Table 5Convergent and Divergent Validities
      MDASI SubscalesPCAS SubscalesFACIT-Sp
      QUAL-E (Fam) Items and SubscalesSymptom SeveritySymptom InterferenceTrustCommunicationComprehensive KnowledgeInterpersonalTotal Score
       Symptom experience−0.27
      P <0.001.
      −0.28
      P <0.001.
      0.05−0.11−0.03−0.050.05
       Symptom severity−0.27
      P <0.001.
      −0.27
      P <0.001.
      0.02−0.03−0.08−0.050.19
      P <0.01.
       Symptom interference−0.34
      P <0.001.
      −0.47
      P <0.001.
      −0.010.070.000.010.17
      P <0.01.
       Symptom total−0.46
      P <0.001.
      −0.44
      P <0.001.
      −0.060.060.130.040.23
      P <0.001.
      Completion (3 items)0.14
      P <0.05.
      0.31
      P <0.001.
      0.04−0.18
      P <0.01.
      −0.09−0.09−0.12
      Relationship with HCP (4 items)0.010.100.59
      P <0.001.
      −0.69
      P <0.001.
      0.54
      P <0.001.
      −0.66
      P <0.001.
      −0.18
      P <0.01.
       Peace0.16
      P <0.05.
      0.26
      P <0.001.
      0.18
      P <0.01.
      −0.15−0.12−0.14
      P <0.05.
      −0.50
      P <0.001.
       Peace care0.090.23
      P <0.01.
      0.55
      P <0.001.
      −0.55
      P <0.001.
      −0.50
      P <0.001.
      −0.61
      P <0.001.
      −0.25
      P <0.001.
       Financial strain−0.18
      P <0.01.
      −0.16
      P <0.05.
      −0.090.010.000.090.09
       Prepared0.110.13
      P <0.05.
      0.040.02−0.04−0.05−0.35
      P <0.001.
      Overall experience−0.010.15
      P <0.05.
      0.72
      P <0.001.
      −0.75
      P <0.001.
      −0.63
      P <0.001.
      −0.79
      P <0.001.
      −0.21
      P <0.01.
      You overall0.18
      P <0.01.
      0.24
      P <0.01.
      0.13−0.16
      P <0.05.
      −0.13−0.16
      P <0.05.
      −0.35
      P <0.001.
      MDASI = M. D. Anderson Symptom Inventory; PCAS = Primary Care Assessment Survey; FACIT-Sp = Functional Assessment of Chronic Illness Therapy—Spiritual Well-Being; QUAL-E = quality of family experience; HCP = health care provider.
      a P <0.05.
      b P <0.01.
      c P <0.001.
      The completion subscale was significantly but weakly associated with the PCAS communication subscale. This is likely because both subscales relate to communication but not with the HCP. There was a divergence between completion and other PCAS subscales, as expected, given that these constructs are unrelated conceptually.
      As expected, peace with care and overall experience with care single items were significantly and strongly associated with all PCAS subscales (all >0.50, P < 0.0001; all >0.60, P < 0.0001, respectively). Other scale items were not expected to correlate with the PCAS and did not, demonstrating discriminant validity.
      The FACIT-Sp showed a significant and strong association with the single item assessing personal sense of peace, thereby demonstrating convergent validity (0.5, P < 0.0001). Single causal items assessing sense of preparation and overall well-being also demonstrated significant but weak association with the FACIT-Sp, as expected (0.35, P < 0.0001, for both). The completion subscale was not significantly associated with spiritual well-being, indicating that these are distinct constructs. Given the association with the PCAS mentioned previously, it is likely tapping a construct more closely associated with communication or personal interaction.
      The MDASI showed significant relationships with items related to symptom experience. In particular, there was a strong association with symptom total (0.46, P < 0.0001), family members' perceptions of patient suffering. There was a significant but weak association between how often the symptom was experienced and how much it interfered with daily life (0.34; P < 0.0001). The MDASI is designed to assess symptom burden through severity, so these responses demonstrate convergent validity.
      There were no associations between symptom items and completion or health care communication scales, demonstrating these as distinct constructs.
      Similar to the QUAL-E and other quality-of-life measures, issues of subscale conceptual distinction prohibit calculation of a total scale score. However, subscale scores are calculable. For the effect indicator subscales, individual scores are summed and averaged. The three questions on completion are summed and divided by three, for a subscale completion total score. The four items on the relationship with HCP are summed and divided by four, for an average relationship total score. The symptom impact and preparation domains are summed as composite scores; these composite scores can be compared between respondents, and threshold values can be used. Individual items also may be examined for clinical relevance, including the single overall items, which were retained individually.
      • Qual-E (Fam) Scoring
      • Symptom impact: Q1 + Q2 + Q3 + Q4.
      • Relationship with HCP: (Q5 + Q6 + Q7 + Q8)/4.
      • Completion: (Q9 + Q10 + Q11)/3.
      • Preparation: Q12 + Q13 + Q14 + Q15.

      Discussion

      In palliative care, the patient and family comprise the unit of care. Therefore, comprehensive assessment necessitates the ability to evaluate the quality of experience for both patients and families. In previous research, we developed the QUAL-E, a measure of patient quality of life. The results of this study yield the companion family measure, the QUAL-E (Fam). It is brief (17 items) (Appendix) and includes items measuring key constructs of the family's relationship with HCPs and completion in their relationship with the patient. Additional individual items assess families' perception of symptom experience and preparation, financial strain, peacefulness, and sense of peace with care. Thus, researchers may assess overall family experience using the total scale or individual constructs.
      Researchers may choose to use the entire QUAL-E (Fam) or use subscales or individual items. We plan to assess its use as a clinical tool as well. For example, we found that a single item from the QUAL-E had clinical value in asking patients the extent to which they were at peace.
      • Steinhauser K.E.
      • Voils C.I.
      • Clipp E.C.
      • et al.
      “Are you at peace?”: one item to probe spiritual concerns at the end of life.
      It showed strong relationships to emotional and spiritual well-being and has been used as a clinical communication tool. Initial analyses of the QUAL-E (Fam) suggesting the item asking family members the extent to which they are at peace with care shows a robust relationship to constructs of trust, interpersonal interaction, knowledge, and communication.
      The QUAL-E (Fam) is one of few available tools to assess the quality of family experience. FAMCARE by Kristjanson
      • Kristjanson L.J.
      Family members' perceptions of palliative cancer care: predictors of family functioning.
      is the oldest available tool used for this purpose and has been used in a variety of end-of-life studies. It includes domains of availability of care, physical care, psychosocial care, and information giving. However, it is limited in several ways. First, it was developed for use in a single population, ambulatory cancer patient families. Second, it is a measure of family satisfaction, and as such, is limited by many of the scaling issues with satisfaction scales, namely poor sensitivity and ceiling effects. Finally, it was validated in a small sample. The FATE, by Casarett et al.,
      • Casarett D.
      • Pickard A.
      • Bailey F.A.
      • et al.
      A nationwide VA palliative care quality measure: the family assessment of treatment at the end of life.
      was developed to assess the postdeath retrospective assessments of family members of quality of care. A 32-item measure, it includes nine subscales of well-being and dignity, communication between family and providers, care around the time of death, emotional and spiritual support, symptom management, respect for treatment preferences, site of death, access to home care services, and access to services after death. It was developed as a conceptual companion to other VA end-of-life quality assessment tools. The QUAL-E assesses patient quality of life at the end of life, the QUAL-E (Fam) assesses family experience at the end of life, and the FATE, quality of care, retrospectively. Together, these allow for a comprehensive measurement strategy for assessing the experiences of dying veterans and their families.

      Limitations

      The data from cognitive interviews and validation samples were limited by the predominance of women. Similar to other studies of caregiving in seriously ill patients, most participants were female. The sample included just under 20% male caregivers, for the validation and cognitive interviews.

      Conclusions

      Family members are part of the unit of care in palliative care, experiencing their own concerns and distress when attending to loved ones at the end of life. The development of the QUAL-E (Fam) was in direct response to this need. The instrument provides an evaluation tool for administrators to assess the quality and effectiveness of interventions seeking to improve end-of-life care. Clinical staff could use the instrument for immediate feedback on aspects of end-of-life care that veterans and their families consider most important. Investigators could use the measurement tool for health services research in this population.

      Disclosures and Acknowledgments

      This material was based on work supported by the Office of Research and Development, Health Services Research and Development Service, Department of Veterans Affairs ( IIR 02–285 ). The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs. The authors declare no conflicts of interest.

      References

        • Aaronson N.K.
        Quality of life assessment in clinical trials: methodologic issues.
        Control Clin Trials. 1989; 10: 195S-208S
        • Cella D.F.
        • Tulsky D.S.
        • Gray G.
        • et al.
        The Functional Assessment of Cancer Therapy scale: development and validation of the general measure.
        J Clin Oncol. 1993; 11: 570-579
        • Cohen S.
        • Mount B.
        Quality of life assessment in terminal illness: defining and measuring subjective well-being in the dying.
        J Palliat Care. 1992; 8: 40-45
        • Cohen S.
        • Mount B.
        • Strobel M.G.
        • et al.
        The McGill Quality of Life Questionnaire: a measure of quality of life for people with advanced disease. A preliminary study of validity and acceptability.
        Palliat Med. 1995; 9: 207-219
        • Ferrans C.
        • Powers M.
        Quality of Life Index: development and psychometric properties.
        Adv Nurs Sci. 1985; 8: 15-24
        • Keall R.M.
        • Butow P.N.
        • Steinhauser K.E.
        • Clayton J.M.
        Discussing life story, forgiveness, heritage, and legacy with patients with life-limiting illnesses.
        Int J Palliat Nurs. 2011; 17: 454-460
        • Steinhauser K.E.
        Measuring end-of-life care outcomes prospectively.
        J Palliat Med. 2005; 8: S30-S41
        • Steinhauser K.E.
        • Bosworth H.B.
        • Clipp E.C.
        • et al.
        Initial assessment of a new instrument to measure quality of life at the end of life.
        J Palliat Med. 2002; 5: 829-841
        • Casarett D.
        • Pickard A.
        • Bailey F.A.
        • et al.
        A nationwide VA palliative care quality measure: the family assessment of treatment at the end of life.
        J Palliat Med. 2008; 11: 68-75
        • Teno J.M.
        • Clarridge B.
        • Casey V.
        • Edgman-Levitan S.
        • Fowler J.
        Validation of Toolkit After-Death Bereaved Family Member Interview.
        J Pain Symptom Manage. 2001; 22: 752-758
        • Connor S.R.
        • Teno J.
        • Spence C.
        • Smith N.
        Family evaluation of hospice care: results from voluntary submission of data via website.
        J Pain Symptom Manage. 2005; 30: 9-17
        • Portenoy J.
        • Teno J.M.
        Hispanic language version of the family evaluation of hospice care.
        J Pain Symptom Manage. 2007; 34: 459-461
        • Rhodes R.L.
        • Mitchell S.L.
        • Miller S.C.
        • Connor S.R.
        • Teno J.M.
        Bereaved family members' evaluation of hospice care: what factors influence overall satisfaction with services?.
        J Pain Symptom Manage. 2008; 35: 365-371
        • Teno J.M.
        Measuring end-of-life care outcomes retrospectively.
        J Palliat Med. 2005; 8: S42-S49
        • Steinhauser K.E.
        • Clipp E.C.
        • Tulsky J.A.
        Evolution in measuring the quality of dying.
        J Palliat Med. 2002; 5: 407-414
        • Bosworth H.B.
        • Steinhauser K.E.
        • Orr M.
        • et al.
        Congestive heart failure patients’ perceptions of quality of life: the integration of physical and psychosocial factors.
        Aging Mental Hlth. 2003; 8: 83-91
        • Steinhauser K.E.
        • Clipp E.C.
        • McNeilly M.
        • et al.
        In search of a good death: observations of patients, families, and providers.
        Ann Intern Med. 2000; 132: 825-832
        • DeMaio T.
        • Rothgeb J.
        Cognitive interviewing techniques: In the lab and in the field.
        Jossey-Bass, San Francisco1996
        • Pfieffer E.
        A short portable mental status questionnaire for the assessment of organic brain deficit in elderly patients.
        J Am Geriatr Soc. 1975; 23: 433-441
        • Armstrong T.S.
        • Mendoza T.
        • Gning I.
        • et al.
        Validation of the M.D. Anderson Symptom Inventory Brain Tumor Module (MDASI-BT).
        J Neurooncol. 2006; 80: 27-35
        • Safran D.G.
        • Kosinski M.
        • Tarlov A.R.
        • et al.
        The Primary Care Assessment Survey: tests of data quality and measurement performance.
        Med Care. 1998; 36: 728-739
        • Safran D.G.
        • Taira D.A.
        • Rogers W.H.
        • et al.
        Linking primary care performance to outcomes of care.
        J Fam Pract. 1998; 47: 213-220
        • Peterman A.H.
        • Fitchett G.
        • Brady M.J.
        • Hernandez L.
        • Cella D.
        Measuring spiritual well-being in people with cancer: the functional assessment of chronic illness therapy–Spiritual Well-being Scale (FACIT-Sp).
        Ann Behav Med. 2002; 24: 49-58
        • Bollen K.
        • Lennox R.
        Conventional wisdom on measurement: a structural equation perspective.
        Psychol Bull. 1991; 110: 305-314
        • Fayers P.M.
        • Hand D.J.
        • Bjordal K.
        • Groenvold M.
        Causal indicators in quality of life research.
        Qual Life Res. 1997; 6: 393-406
        • Voils C.I.
        • Hoyle R.H.
        • Thorpe C.T.
        • Maciejewski M.L.
        • Yancy Jr., W.S.
        Improving the measurement of self-reported medication nonadherence.
        J Clin Epidemiol. 2011; 64: 250-254
        • Bentler P.
        Comparative fit indices in structural models.
        Psychol Bull. 1990; 107: 238-246
        • Bentler P.
        • Bonett D.
        Significance tests and goodness-of-fit in the analysis of covariance structures.
        Psychol Bull. 1980; 88: 588-606
        • Joreskog K.
        • Sorbom D.
        • Lisrel V.
        Analysis of linear structural relationships by the method of maximum likelihood.
        National Educational Resources, Chicago1981
      1. Steiger J, Lind J. Statistically based tests for the number of common factors. Paper presented at the annual meeting of the Psychometric Society, Iowa City, IA, 1980.

        • Browne M.
        • Cudeck R.
        Alternative ways of assessing model fit.
        Sage, Newbury Park1993
        • Steinhauser K.
        • Clipp E.
        • Bosworth H.
        • et al.
        Measuring quality of life at the end of life: validation of the QUAL-E.
        J Palliat Support Care. 2004; 2: 3-14
        • Steinhauser K.E.
        • Voils C.I.
        • Clipp E.C.
        • et al.
        “Are you at peace?”: one item to probe spiritual concerns at the end of life.
        Arch Intern Med. 2006; 166: 101-105
        • Kristjanson L.J.
        Family members' perceptions of palliative cancer care: predictors of family functioning.
        J Palliat Care. 1996; 12: 10-20