Advertisement

Validation of ‘Care Of the Dying Evaluation’ (CODETM) within an international study exploring bereaved relatives’ perceptions about quality of care in the last days of life

Open AccessPublished:March 04, 2022DOI:https://doi.org/10.1016/j.jpainsymman.2022.02.340

      Abstract

      Context

      Assessing quality of care provided during the dying phase using validated tools aids quality assurance and recognizes unmet need.

      Objective

      To assess construct validity and internal consistency of ‘Care Of the Dying Evaluation’ (CODETM) within an international context.

      Methods

      Post-bereavement survey (August 2017 to September 2018) using CODETM. Respondents were next-of-kin to adult patients (≥ 18 years old) with cancer who had an ‘expected’ death within 22 study site hospitals in 7 countries: Argentina, Brazil, Germany, Norway, Poland, United Kingdom, Uruguay. Exploratory and Confirmatory Factor Analysis (EFA and CFA) were conducted, and internal reliability was assessed using Cronbach alpha (α). Known group validity was assessed by ability to discriminate quality of care based in place (Palliative Care Units (PCUs)) and country (Poland, where most deaths were in PCUs) of care. Differences were quantified using effect sizes (ES).

      Results

      A 914 CODETM questionnaires completed (54% response rate). 527 (58%) male deceased patients; 610 (67%) next-of-kin female who were most commonly the ‘spouse/partner’ (411, 45%). EFA identified 4 factors: ‘Overall care,’ ‘Communication and support,’ ‘Trust, respect and dignity,’ and ‘Symptom management’ with good reliability scores (α = 0.628 – 0.862). CFA confirmed the 4-factor model; these were highly correlated and a bifactor model showed acceptable fit. The ES for quality of care in PCU's was 0.727; ES for Poland was 0.657, supporting the sensitivity of CODETM to detect differences.

      Conclusion

      Within an international context, good evidence supports the validity and reliability of CODETM for assessing the quality of care provided in the last days of life.

      Key Words

      Key Message

      This article describes the validation of the ‘Care Of the Dying Evaluation’ (CODETM) questionnaire within an international study. The results indicate that CODETM represents a valid and reliable tool for assessing the quality of care provided to dying patients and their families.

      Introduction

      Ensuring dying patients and their families receive high quality care and support is fundamentally important. How an individual dies has a profound impact on those bereaved. Timely, informative end-of-life discussions are associated with less complicated grief.
      • Yamaguchi T
      • Maeda I
      • Hatano Y
      • et al.
      Effects of end-of-life discussions on the mental health of bereaved family members and quality of patient death and care.
      Conversely, overly aggressive medical interventions and dissatisfaction with communication are linked with poorer bereavement outcomes.
      • Wright AA
      • Zhang B
      • Ray A
      • et al.
      Associations between end-of-life discussions, patient mental health, medical care near death, and caregiver bereavement adjustment.
      ,
      • Miyajima K
      • Fujisawa D
      • Yoshimura K
      • et al.
      Association between quality of end-of-life care and possible complicated grief among bereaved family members.
      Globally, the provision of care for dying patients varies, as indicated by the Quality of Death Index.

      The Economist Intelligence Unit. The 2015 quality of death index. Lien Foundation. Available at:https://www.eiuperspectives.economist.com/sites/default/files/2015%20EIU%20Quality%20of%20Death%20Index%20Oct%2029%20FINAL.pdf. Accessed November 16, 2021.

      From a clinical and research perspective, it is important to have valid and reliable measures to assess the current quality of care. One method of evaluation is to use validated tools and ask family members, or those deemed important to the deceased, about their experiences. An example of such a tool is the ‘Care Of the Dying Evaluation’ (CODETM) questionnaire. CODETM is a shortened version of the original, validated instrument, ‘Evaluating Care and Health Outcomes – for the Dying (ECHO-D).
      • Mayland CR
      • Williams EM
      • Ellershaw JE.
      Assessing quality of care for the dying: the development and initial validation of a postal self-completion questionnaire for bereaved relatives.
      • Mayland CR
      • Williams EM
      • Addington-Hall J
      • et al.
      Does the ‘Liverpool Care Pathway’ facilitate an improvement in quality of care for dying cancer patients?.
      • Mayland CR
      • Williams EM
      • Addington-Hall J
      • et al.
      Assessing the quality of care for dying patients from the bereaved relatives' perspective: further validation of “Evaluating care and health outcomes–for the dying”.
      Both tools are unique as their conceptual basis relates to the key components of best practice for ‘care for the dying’ in the last days of life.
      • Mayland C
      • Williams E
      • Ellershaw J.
      How well do current instruments using bereaved relatives' views evaluate care for dying patients?.
      CODETM assesses the quality of patient care and the level of family support, through 32 main questions, reflecting core palliative care principles. Ten additional questions focus on demographic details.
      CODETM was initially validated in the United Kingdom (UK) within the community setting
      • Mayland CR
      • Lees C
      • Germain A
      • et al.
      Caring for those who die at home: the use and validation of ‘Care Of the Dying Evaluation’ (CODE) with bereaved relatives.
      and has subsequently been used across UK hospital,
      • Mayland CR
      • Mulholland H
      • Gambles M
      • et al.
      How well do we currently care for our dying patients in acute hospitals: the views of the bereaved relatives?.
      ,
      • Vogt A
      • Stiel S
      • Heckel M
      • et al.
      Assessment of the quality of end-of-life care: translation and validation of the German version of the “Care of the Dying Evaluation” (CODE-GER) - a questionnaire for bereaved relatives.
      hospice and home settings.
      • Mayland C
      • McGlinchey T
      • Gambles M
      • et al.
      Quality assurance for care of the dying: engaging with clinical services to facilitate a regional cross-sectional survey of bereaved relatives' views.
      Within a systematic review of 67 tools used after death, CODETM was one of the four recommended for use, based on initial psychometric properties.
      • Kupeli N
      • Candy B
      • Tamura-Rose G
      • et al.
      Tools measuring quality of death, dying, and care, completed after death: systematic review of psychometric properties.
      Assessment of CODETM in an international setting is therefore necessary to evaluate its robustness in the wider context of care for the dying.
      The aim of this study was to evaluate the psychometric properties of CODETM in an international context. The two objectives were to assess the:
      • 1.
        Construct validity of CODETM using factor analysis techniques and undertaking international country comparisons to evaluate any differences in perceptions.
      • 2.
        Internal consistency of the tool.

      Methods

      Details of study design and participants have been comprehensively described,
      • Haugen DF
      • Hufthammer KO
      • Gerlach C
      • et al.
      ERANet-LAC CODE Project Group. good quality care for cancer patients dying in hospitals, but information needs unmet: bereaved relatives' survey within seven countries.
      hence a summary is provided below. For clarity, ‘next-of-kin’ is a collective term for family members, friends and neighbors.

      Participants

      Respondents were the next-of-kin to adult cancer patients (≥ 18 years old) who had died an ‘expected’ death within the study site hospitals (n = 22) in seven South American and European countries: Argentina, Brazil, Germany, Norway, Poland, UK, and Uruguay. The patient must have been admitted to the hospital for at least three calendar days. The next-of-kin were eligible to complete the survey if aged ≥ 18 years, sufficiently fluent in the language, and able to provide informed consent. This was pragmatically assessed by ward staff at the time of death and by research staff directly contacting potential participants to invite them to participate.

      Instrument and Development

      Work was conducted to develop CODETM into an international tool (i-CODE) involving forward-and-back translation within each of the five different languages.
      • Mayland CR
      • Gerlach C
      • Sigurdardottir K
      • et al.
      Assessing quality of care for the dying from the bereaved relatives' perspective: Using pre-testing survey methods across seven countries to develop an international outcome measure.
      Pre-testing survey methods, involving patient and public representatives and bereaved relatives, and including cognitive interviews, helped ensure good face and content validity.
      • Mayland CR
      • Gerlach C
      • Sigurdardottir K
      • et al.
      Assessing quality of care for the dying from the bereaved relatives' perspective: Using pre-testing survey methods across seven countries to develop an international outcome measure.
      Consensus about the tool's content was reached using a modified nominal group technique.
      • Jones J
      • Hunter D.
      Consensus methods for medical and health services research.
      This established a core, collective international version of CODETM (i-CODE, Supplementary File 1) to use within the seven countries. Response options include Likert scale verbal anchors and ordinal responses where higher values represent better quality of care. As i-CODE contains all the questions from the original CODETM questionnaire (with additional questions about advance care planning and the NHS Friends and Family test

      National Health Service. Friends and family test. Available at: https://www.england.nhs.uk/fft. Accessed September 25, 2021.

      being added by some countries), we use the terminology ‘CODETM’ within this paper.

      Procedure

      Data was collected between August 15, 2017 and September 15, 2018. A postal survey was planned but different approaches were adopted to reflect country-specific factors such as unreliable postal services and literacy issues. Poland, Argentina, Brazil and Uruguay undertook face-to-face or telephone interviews.
      Screening for eligible cases was undertaken by the research team and information on the deceased patient's gender, age, cancer, length of hospital stay and place of death (type of hospital ward) as well as next-of-kin's gender and age group were collected.
      Following screening for eligibility, the CODETM questionnaire was sent or administered to next-of-kin 6-8 weeks after the patient's death. Responses were entered into a database and data were stored on a protected research server.

      Statistical Analysis

      Correlation

      Polychoric correlation was used to assess relationships of items prior to factor analysis; items with correlation coefficient ≥ 0.8 indicated local dependency. Item wording was also considered to identify any potential overlap. This informed which items to combine or drop in the analysis with supplementary analysis of the excluded questions.

      Exploratory and Confirmatory Factor Analyses

      Previous work on the dimensionality of CODETM has revealed different structures and included different items.
      • Mayland CR
      • Lees C
      • Germain A
      • et al.
      Caring for those who die at home: the use and validation of ‘Care Of the Dying Evaluation’ (CODE) with bereaved relatives.
      ,
      • Vogt A
      • Stiel S
      • Heckel M
      • et al.
      Assessment of the quality of end-of-life care: translation and validation of the German version of the “Care of the Dying Evaluation” (CODE-GER) - a questionnaire for bereaved relatives.
      Therefore, we carried out an Exploratory Factor Analysis (EFA) followed by Confirmatory Factor Analysis (CFA), using Geomin rotation to identify potential factors present in the data. EFA and CFA were performed treating the items as ordinal categorical, using the robust weighted least squares means and variance adjusted (WLSMV) estimator in Mplus 8.2.
      • Flora DB
      • Curran PJ.
      An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data.
      ,
      • Muthen LK
      • Muthen BO.
      Mplus Users Guide (Sixth ed.).
      Model fit was assessed by the Root Mean Square Error of Approximation (RMSEA)
      • MacCallum RC
      • Browne MW
      • Sugawara HM.
      Power analysis and determination of sample size for covariance structure modeling.
      and the Comparative Fit Index (CFI)
      • Hu L-t
      • Bentler PM.
      Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives.
      where a value of ≤.08 and >.95 was assumed to provide a good fit respectively. To obtain finer factor solutions, residual correlations and modification indices (MI) were inspected to identify potentially redundant items.
      • Schreiber JB
      • Stage FK
      • King J
      • et al.
      Reporting structural equation modeling and confirmatory factor analysis results: a review.
      ,
      • Watt T
      • Groenvold M
      • Deng N
      • et al.
      Confirmatory factor analysis of the thyroid-related quality of life questionnaire ThyPRO.
      In the final models, local dependence between items was introduced guided by:
      • i)
        the highest MI (>100)
        • Watt T
        • Groenvold M
        • Deng N
        • et al.
        Confirmatory factor analysis of the thyroid-related quality of life questionnaire ThyPRO.
        or
      • ii)
        whether the pairs of items had been identified as conceptually similar, by constraining the pair of items as free parameter estimates in model revisions, one at a time.

      Internal Reliability of the Scale

      Internal reliability was assessed using Cronbach alpha to quantify the extent to which the items were inter-related. Coefficients above 0.7 are acceptable, above 0.8 are good, and above 0.9 are excellent but above 0.94 suggests potential redundancy.
      • Fayers PM
      • Machin D.
      Quality of life: the assessment, analysis and interpretation of patient-reported outcomes.

      Known-Group Validity

      Known group validity was examined in terms of whether CODETM was able to discriminate between the different quality of care expected a priori. First, we hypothesized that the quality of care would be higher in hospital Palliative Care Units (PCUs) compared with other hospital wards. Second, we hypothesized that as most deaths in Poland occurred in hospital PCUs, we expected perceptions about quality of care to be higher. Differences were quantified using effect sizes (ES) across categories identified, calculated as the difference in mean scores between groups divided by the standard deviation of the lower quality of the two sub-groups. ES expressed as Cohen's d of 0.2 are normally considered small, 0.5 moderate, and 0.8 large.
      • Cohen J.
      Statistical power analysis for the behavior science.

      Testing for Item Invariance

      Differential item functioning (DIF) is present when participants with the same score level (quality of care, in this instance) endorse items differently by virtue of some characteristics other than the variation due to their scale score. The simple sum of the items in question was used as a proxy for the latent trait. Uniform DIF exists when the statistical relationship between item response and group is constant for all levels of a matching variable. Non-uniform DIF exists when there are differences among the groups for specific item responses.
      • Hanson BA.
      Uniform DIF and DIF Defined by Differences in Item Response Functions.
      DIF with regard to age and gender of patient and relative, ward and country was evaluated through ordinal logistic regression models.
      • Zumbo BD.
      A handbook on the theory and methods of differential item functioning (DIF).
      Significant DIF was assessed through a dual criterion of statistical significance and a difference in explained variance (Nagelkerke pseudo R2) larger than 2%.
      • Nagelkerke NJ.
      A note on a general definition of the coefficient of determination.
      To assess the impact of the different countries on the factor structure, we regressed the factors on country in the best performing CFA model.

      Missing Data

      Missing data for the items was less than 5% and no data imputation was conducted.

      Results

      Response Rate

      From 1683 potential cases screened, 914 CODETM questionnaires were completed (54% response rate) with at least 100 responses per country.

      Demographics (Table 1)

      The deceased patients tended to be male (527, 58%) and the most common primary cancer diagnoses were from the gastrointestinal tract (321, 35%) or respiratory system (196, 21%). Next-of-kin tended to be female (610, 67%) and the ‘spouse and/or partner’ (411, 45%) to the deceased individual. Further demographics have been detailed previously.
      • Haugen DF
      • Hufthammer KO
      • Gerlach C
      • et al.
      ERANet-LAC CODE Project Group. good quality care for cancer patients dying in hospitals, but information needs unmet: bereaved relatives' survey within seven countries.

      Correlation Results

      High correlations (0.8–0.99) were observed for a number of questions and further examination of wording revealed considerable overlap (Table 2).
      Table 1Demographic Details of the Deceased Individuals and Participants (n = 914)
      Deceased individualParticipants
      N%
      Percentage. Missing data presented but not included in the percentage calculations.
      N%
      Percentage. Missing data presented but not included in the percentage calculations.
      Gender
       Male52758%29833%
       Female38742%61067%
      (Missing)00%61%
      Age group
       Under 50556%26930%
       51–7965171%59966%
       80+20823%354%
      (Missing)00%111%
      Relationship to deceased patient
       Spouse/partner41145%
       Son/daughter31735%
       Brother/sister677%
       Son-in-law/daughter-in-law233%
       Parent152%
       Friend/neighbour758%
       (Missing)61%
      Cancer diagnosis
      16 patients registered with two cancer types and 2 patients with three cancer types.
       Gastrointestinal, incl. pancreatic32135%
       Respiratory organs19621%
       Leukemia/lymphoma10011%
       Urological, incl. prostate839%
       Breast435%
       Brain344%
       Gynecological334%
       Other12213%
      (Missing)20%
      Type of hospital ward where the patient died
       Medical or surgical ward44749%
       Palliative care unit23125%
       Oncology ward11913%
       Intensive care unit698%
       Emergency unit425%
      (Missing)6<1%
      a Percentage. Missing data presented but not included in the percentage calculations.
      b 16 patients registered with two cancer types and 2 patients with three cancer types.
      Table 2Question Items with High Correlations and Implications for Analysis
      ItemWordingImplication for analysis
      High correlation items (r = 0.8-0.99)
      q17Did any of the healthcare team discuss with you whether giving fluids through a ‘drip’ would be appropriate in the last two days of life?Combined q17 and q18 as they are related
      q18Would a discussion about the appropriateness of giving fluids through a ‘drip’ in the last two days of life have been helpful?
      q21Overall, his/her religious or spiritual needs were met by the healthcare team.Kept q21 pertaining to the spiritual needs of the patient rather than the family carer as the foremost responsibility of the healthcare organisation is to the patient. The EFA were repeated with dropping q21 and keeping q22 to assess any differences (none found).
      q22Overall, my religious or spiritual needs were met by the healthcare team.
      q24Did a member of the healthcare team talk to you about what to expect when he/she was dying (eg, symptoms that may arise)?Combined q24 and q25 as they are related
      q25Would a discussion about what to expect when he/she was dying have been helpful?
      Further examination of wording of items showing overlap
      q10In your opinion, during the last two days, did he/she appear to be in pain?Excluded q10
      q11In your view, did the doctors and

      nurses do enough to help relieve the

      pain?
      q12In your opinion, during the last two days, did he/she appear to be restless?Excluded q12
      q13In your view, did the doctors and nurses do enough to help relieve the restlessness?
      q14In your opinion, during the last two days, did he/she appear to have a ‘noisy rattle’ to his/her breathing?Excluded q14
      q15In your view, did the doctors and nurses do enough to help relieve the ‘noisy rattle’ to his/her breathing?

      Exploratory Factor Analysis Results

      Eigenvalue analysis identified a strong first factor and four or five weaker factors with values >1 (Supplementary Figure S1, Supplementary Table S1/S2). EFA resulted in a four-factor model, namely: ‘Overall care,’ ‘Communication and support,’ ‘Trust, respect and dignity,’ and ‘Symptom management.’

      Estimating Four-Factor and Bi-Factor CFA Models

      The four-factor model returned an acceptable fit (RMSEA = 0.069, CFI = 0.939). The standardized coefficients of the four factors ranged from 0.366 to 0.915 and were all statistically significant (P < 0.001). The correlations between factors ranged between 0.656 and 0.858 (Table 3). We extended the four-factor model to account for areas of strain within factor solutions through local correlations. Two pairs of items were allowed to correlate with each other: q1 (personal care needs) and q2 (nursing care needs); q3 (environment was comfortable) and q4: (environment had adequate privacy). The model fit improved (RMSEA = 0.062; CFI = 0.951) with the factors still highly correlated (Table 3) leading us to estimate a bi-factor model. The model achieved acceptable fit (RMSEA = 0.057; CFI = 0.962) and the loadings for the global factor are higher than for the individual factors for the majority of items. Explained common variance (ECV) values were 77.9%, showing a strong global factor and suggesting a unidimensional model.
      • Reise S.P.
      • Moore T.M.
      • Haviland M.G.
      Bifactor models and rotations: Exploring the extent to which multidimensional data yield univocal scale scores.
      Table 3Parameter Estimates for the Four-Factor and Bi-Factor Models
      Question item4 factor model (4cfa2) MIBi-factor with 4 factors with MI
      F1F2F3F4GlobalF1F2F3F4
      Overall careCommunication & supportTrust, respect & dignitySymptom managementOverall careCommunication & supportTrust, respect & dignitySymptom management
      q1There was enough help available to meet his/her personal care needs, such as washing, personal hygiene and toileting needs.0.7860.6990.337
      q2There was enough help with nursing care, such as giving medicines and helping him/her find a comfortable position in bed0.8210.7140.412
      q3The bed area and surrounding environment was comfortable for him/her0.720.5910.653
      q4The bed area and surrounding environment had adequate privacy for him/her0.6540.5490.538
      q5In your opinion, how clean was the ward area that he/she was in?0.6190.5570.254
      q6Did you have confidence and trust in the nurses who were caring for him/her?0.8310.760.23
      q7Did you have confidence and trust in the doctors who were caring for him/her?0.8420.7680.247
      q8The nurses had time to listen and discuss his/her condition with me.0.7940.7480.166
      q9The doctors had time to listen and discuss his/her condition with me.0.810.7960.038
      q11In your view, did the doctors and nurses do enough to help relieve the pain?0.9150.7040.337
      q13In your view, did the doctors and nurses do enough to help relieve the restlessness?0.7670.5740.748
      q15In your view, did the doctors and nurses do enough to help relieve the ‘noisy rattle’ to his/her breathing?0.5390.390.45
      q16During the last two days, how involved were you with the decisions about his/her care and treatment?0.5210.450.429
      q1718Did any of the healthcare team discuss with you whether giving fluids through a ‘drip’ would be appropriate in the last two days of life?0.3660.3290.22
      q19Did the healthcare team explain his/her condition and/or treatment in a way you found easy or difficult to understand?0.6740.6110.4
      q20How would you assess the overall level of emotional support given to you by the healthcare team?0.7770.7180.382
      q21Overall, his/her religious or spiritual needs were met by the healthcare team.0.6250.5770.277
      q23Before he/she died, were you told he/she was likely to die soon?0.3950.3040.558
      q2425Did a member of the healthcare team talk to you about what to expect when he/she was dying (e.g. symptoms that may arise)?0.4820.3950.534
      q27In your opinion did he/she die in the right place?0.6450.5790.262
      q28I was given enough help and support by the healthcare team at the actual time of his/her death.0.8130.7690.235
      q29After he/she had died, did individuals from the healthcare team deal with you in a sensitive manner?0.5390.4910.153
      q30dHow much of the time was he/she treated with respect and dignity in the last two days of life? (By doctors)0.7730.6880.37
      q30nHow much of the time was he/she treated with respect and dignity in the last two days of life? (By nurses)0.810.7250.369
      q31Overall, in your opinion, were you adequately supported during his/her last two days of life?0.7410.7010.21
      q32How likely are you to recommend our Organisation to friends and family?0.7570.7140.153
      Correlations between factorsCorrelations between factors
      the correlations between the factors are all set to zero as required by this specification of the bi-factor model.
      F2 (Communication & support) and F1 (Overall care)0.858Global with F1, F2, F3, F40
      F3 (Trust, respect and dignity) and F1 (Overall care)0.777F1 (Overall care) with F2 (Communication & support)0
      F3 (Trust, respect and dignity) and F2 (Communication & support)0.833F1 (Overall care) with F3 (Trust, respect and dignity)0
      F4 (Symptom management) and F1 (Overall care)0.656F1 (Overall care) with F4 (Symptom management)0
      F4 (Symptom management) and F2 (Communication & support)0.678F2 (Communication & support) with F3 (Trust, respect and dignity)0
      F4 (Symptom management) and F3 (Trust, respect and dignity)0.747F2 (Communication & support)with F4 (Symptom management)0
      F3 (Trust, respect and dignity)with F4 (Symptom management)0
      Correlations between questions

      (introduced because of high MI)
      Correlations between questions

      (introduced because of high MI)
      Q1 (personal care needs) and Q2 (nursing care)0.198Q1 (personal care needs) and Q2 (nursing care)0.205
      Q3 (environment comfortable) and Q4 (privacy of environment)0.212
      MI = modification indices.
      a the correlations between the factors are all set to zero as required by this specification of the bi-factor model.

      Reliability Results

      The four-factor scale showed moderate to excellent reliability scores (Factor 1 ‘Overall care’ α = 0.862; Factor 2 ‘Communication and support’ α = 0.824; Factor 3 ‘Trust, respect and dignity’ α = 0.618; Factor 4 ‘Symptom management’ α = 0.796; Overall α = 0.922).

      Known-Group Validity

      The ES for the quality of care in PCUs and for Poland were 0.727 and 0.657 respectively. In both cases, the results are as hypothesized, suggesting a higher quality of care in PCUs and in Poland compared with other countries.

      Testing for Item Invariance

      In tests of DIF with regard to age, gender, ward and country, several statistically significant instances of DIF were found (Table 4, Supplementary Tables S3a-S6). Uniform DIF was found for q4 for patient age and q27 and q29 for relative age. There was uniform DIF for q5 for relative gender. The largest number of instances of DIF was seen with regard to country. Perceptions about many aspects of care were often higher from Polish respondents for example, symptom control, support after death. Additionally, perceptions about aspects of nursing care in Argentina had lower mean scores (Table 5).
      Table 4Differential Item Functioning (DIF) Results
      Items not exhibiting DIF are not presented in this table; (1) uniform DIF (2) non-uniform DIF ✗ DIF present.
      Age (patient)Age (relative)Gender (relative)CountryWard type
      (1)(2)(1)(2)(1)(2)(1)(2)(1)(2)
      q4: The bed area and surrounding environment had adequate privacy for him/her.
      q5: In your opinion, how clean was the ward area that he/she was in?
      Q6: Did you have confidence and trust in the nurses who were caring for him/her?
      q8: The nurses had time to listen and discuss his/her condition with me.
      q13: In your view, did the doctors and nurses do enough to help relieve the restlessness?
      q15: In your view, did the doctors and nurses do enough to help relieve the ‘noisy rattle’ to his/her breathing?
      q17/18: Did any of the healthcare team discuss with you whether giving fluids through a ‘drip’ would be appropriate in the last two days of life?
      q21: Overall, his/her religious or spiritual needs were met by the healthcare team.
      q23: Before he/she died, were you told he/she was likely to die soon?
      q24/25: Did a member of the healthcare team talk to you about what to expect when he/she was dying (e.g. symptoms that may arise)?
      q27: In your opinion did he/she die in the right place?
      q29: After he/she had died, did individuals from the healthcare team deal with you in a sensitive manner?
      a Items not exhibiting DIF are not presented in this table; (1) uniform DIF (2) non-uniform DIF ✗ DIF present.
      Table 5Mean Scores for Selected Items where Uniform DIF has been found with Respect to the Country
      Countyq4
      Non-uniform
      The bed area and surrounding environment had adequate privacy for him/her
      q5
      Uniform.
      In your opinion, how clean was the ward area that he/she was in?
      q6
      Uniform.
      Did you have confidence and trust in the nurses who were caring for him/her?
      q8
      both non-uniform and uniform.
      The nurses had time to listen and discuss his/her condition with me.
      q13
      Uniform.
      In your view, did the doctors and nurses do enough to help relieve the restlessness?
      q15
      Uniform.
      In your view, did the doctors and nurses do enough to help relieve the ‘noisy rattle’ to his/her breathing?
      q21
      Uniform.
      Overall, his/her religious or spiritual needs were met by the healthcare team.
      q23
      Uniform.
      Before he/she died, were you told he/she was likely to die soon?
      q29
      Uniform.
      After he/she had died, did individuals from the healthcare team deal with you in a sensitive manner?
      Argentina2.6291.4481.4622.4421.7141.6672.4000.7880.904
      Brazil3.2571.5451.5902.9711.7331.7142.9520.7880.902
      Germany3.5171.6481.7573.2741.7321.7203.2450.8460.965
      Norway3.4641.3991.7013.3301.7021.6122.5500.8950.971
      Poland3.4201.8691.9493.6501.9001.9193.1900.8901.000
      UK2.7251.6701.6963.1571.5941.6332.5300.8700.958
      Uruguay2.8481.4481.7042.9441.8321.8712.7120.9601.000
      a Uniform.
      b Non-uniform
      c both non-uniform and uniform.

      Discussion

      Within an international context, CODETM was found to be valid and reliable in assessing the quality of care provided in the last days of life for those dying from cancer in hospital. Assessment of construct validity identified that a bi-factor model with four distinct factors – ‘overall care,’ ‘communication and support,’ ‘trust, respect and dignity,’ and ‘symptom management’ - provided the best model fit. For 27 of the 32 questions, the factor loadings were substantially higher on the general factor than on the group factors. Each of the four identified factors had good construct validity and internal consistency. Additionally, our a priori hypotheses were supported by our findings: CODETM was sufficiently sensitive to detect differences in perceptions of care between countries; and the quality of care provided within PCUs was perceived to be greater compared with other ward settings. This study further builds on the quality of psychometric evidence
      • Kupeli N
      • Candy B
      • Tamura-Rose G
      • et al.
      Tools measuring quality of death, dying, and care, completed after death: systematic review of psychometric properties.
      for CODETM, namely for: internal consistency, hypothesis testing and cross-cultural properties.
      The original structure of CODETM is around three composite scales ‘Environment,’ ‘Care’ and ‘Communication.’ These scales did not include all question items but were based upon theoretical assumptions developed and validated from the ‘ECHO-D’ questionnaire.
      • Mayland CR
      • Lees C
      • Germain A
      • et al.
      Caring for those who die at home: the use and validation of ‘Care Of the Dying Evaluation’ (CODE) with bereaved relatives.
      Vogt et al 2020 found a 7-factor model based on principal component analysis of a selected 28 ‘core items’ with some items loading on more than one factor.
      • Vogt A
      • Stiel S
      • Heckel M
      • et al.
      Assessment of the quality of end-of-life care: translation and validation of the German version of the “Care of the Dying Evaluation” (CODE-GER) - a questionnaire for bereaved relatives.
      These differences may be due to methods or samples used and the items included in the final models. Our four identified factors represent meaningful concepts reflecting principles of holistic palliative care. Previous studies have concluded that scales with similar ECV values (77.9%) are sufficiently unidimensional.
      • Reise SP
      • Bonifay WE
      • Haviland MG.
      Scoring and modeling psychological measures in the presence of multidimensionality.
      ,
      • Keetharuth AD
      • Bjorner JB
      • Barkham M
      • et al.
      Exploring the item sets of the Recovering Quality of Life (ReQoL) measures using factor analysis.
      Within a recognized multi-faceted concept, unidimensionality implies that an overall score for ‘quality of care for the dying’ could be calculated for CODETM should this be desired.
      One of the key objectives was to undertake international country comparisons to evaluate for differences in perceptions, which was observed for nine question items. In this context, significant DIF suggests that respondents are answering the questions differently by virtue of being in different countries. These differences may be a tool artefact or may be picking up real differences in perceptions about the quality of care in different countries. For example, within the Argentinian study sites, issues relating to inadequate nursing numbers may have influenced perceptions about these aspects of care.
      • Rubinstein A
      • Zerbino MC
      • Cejas C
      • et al.
      Making universal health care effective in argentina: a blueprint for reform.
      This compares with findings from Poland where most deaths occurred within hospital PCUs and perceptions about nursing care were higher. In these instances, we can conclude that DIF may not actually be problematic but reflects true differences in perceptions.
      Almost 70 tools have been identified to assess quality of death, dying and care with variable levels of use and psychometric assessment.
      • Kupeli N
      • Candy B
      • Tamura-Rose G
      • et al.
      Tools measuring quality of death, dying, and care, completed after death: systematic review of psychometric properties.
      Choice of tool is influenced by many factors including study purpose and setting, the measure's validity, reliability, and ease of use. CODETM as a tool has certain strengths. The content and format are acceptable and relevant across several countries reflecting different cultures. CODETM has the potential to assess interventions used within the last days of life, as well as those used earlier in the disease trajectory. Additionally, CODETM can facilitate quality improvement work, with direct user-feedback helping inform clinical practice.
      • Goldraij G
      • Tripodoro VA
      • Aloisio M
      One chance to get it right: improving clinical handovers for better symptom control at the end of life.
      For statistical purposes, we have dropped some items. While this does not necessarily mean that these items are not helpful, there may be scope to create a shortened version of CODETM. Shortened tools can be a more feasible way to assess quality of care for larger populations,

      Pérez-Cruz PE, Padilla Pérez O, Bonati P, et al. Validation of the Spanish Version of the Quality of Dying and Death Questionnaire (QODD-ESP) in a Home-Based Cancer Palliative Care Program and Development of the QODD-ESP-12. J Pain Symptom Manage. 2017;53:1042-1049 .e3.https://doi.org/10.1016/j.jpainsymman.2017.02.005. Epub 2017 Mar 16. PMID: 28323080.

      may reduce completion burden, and encourage higher response rates.
      Our study has several limitations. It was only conducted with the next-of-kin who had experienced a hospital death related to cancer. We could not conduct a multilevel EFA and CFA because there were only seven ‘clusters’ (countries) and a minimum of 30–50 is required for such a technique. Additional psychometric work using an independent dataset, reflecting a different population is needed to undertake further CFA of the constructs identified within this study. Future work will include testing inter-rater reliability and the concurrent and predictive validity of CODETM. Additionally, using CODETM with other instrument(s) is needed to assess criterion validity, although a pragmatic choice is required as no current ‘gold standard’ tool exists.

      Conclusion

      This study provides good evidence for the validity and reliability of CODETM within several different countries representing different cultural contexts. Using CODETM provides part of the means to ensure unmet needs are recognized and that efforts are made to ensure quality of care provided to dying patients and their families is at the level of the best.

      Disclosures and Acknowledgments

      We have no conflicts of interest to declare.
      We wish to thank all the bereaved family members and friends for taking their time to complete the CODETM questionnaire and sharing their views. Additionally, we wish to thank our wider ERANet-LAC CODETM Project Group:
      Core scientific group: Katrin Ruth Sigurdardottir, Marit Irene Tuen Hansen, Karl Ove Hufthammer, Wojciech Leppert, Katarzyna Wolszczak, Eduardo Garcia Yanneo, Vilma Tripodoro, Gabriel Goldraij, Martin Weber, Christina Gerlach, Lair Zambon, Juliana Nalin Passarini, Ivete Bredda Saad, John Ellershaw, and Grace Ting.
      Local study coordinators: Anne Kvikstad, Eva Gravdahl, Julia Bratke, Janet Bakken, Kristin Vassbotn Guldhav, Christina Demmerle.
      We also thank our colleagues within the International Collaborative for Best Care for the Dying Person and the White Rose collaboration for valuable discussions and support.
      And through the ERANet-LAC CODETM project: “International Care Of the Dying Evaluation (CODETM): quality of care for cancer patients as perceived by bereaved relatives” (reference ELAC2015/T07–0545, January 2017–January 2020) as one of the successful proposals of the second Joint Call for Transnational Research and/or Innovation Projects within the ERANet-LAC framework, cofunded by the European Commission's seventh Framework Programme, with the overall aims to improve the quality of care and quality of life of patients dying from cancer.
      The following national funding agencies are acknowledged: Ministry for Science, Technology and Productive Innovation, Argentina; Research Support Foundation of the State of Sao Paulo, Brazil; Federal Ministry of Education and Research, Germany; The Research Council of Norway (grant number 271051); National Centre for Research and Development, Poland; and Marie Curie and the Northwest Coast Clinical Research Network Funding, U.K. Dr. Catriona Mayland is funded by Yorkshire Cancer Research. The funders of the study played no role in the study design, data collection, data analysis, data interpretation, or writing of the report.

      Ethics and Research Governance

      Study approval was given by the Regional Committee for Medical and Health Research Ethics West (2017/640/REK vest), Norway; Ethics Committee of the Medical Council of the province of Rhineland-Palatine, Germany (approval nos. 837.331.13(901 6F) and 837.292.17 (111261); Guía de Buenas Prácticas de Investigación Clínica en Seres Humanos, Ministerio de Salud de la Nación Argentina (Resolución 1480/2011); Bioethics Committee, Poland - KB507/2017 (13.06.17); Health Research Authority and East of England – Cambridge East Research Ethics Committee, U.K. (IRAS project ID 225922; REC 17/EE/0302); National Research Ethics Commission, Brazil (ref. 2308.216); and Committee on Bioethics from the “Mutualista Asociacion Hospital Evangelico,” Uruguay (ref. 29/5/2017/01).
      The study was registered on ClinicalTrials.gov -Identifier: NCT03566732

      Appendix. Supplementary materials

      References

        • Yamaguchi T
        • Maeda I
        • Hatano Y
        • et al.
        Effects of end-of-life discussions on the mental health of bereaved family members and quality of patient death and care.
        J Pain Symptom Manage. 2017; 54 (e1Epub 2017 Apr 24. PMID: 28450216): 17-26https://doi.org/10.1016/j.jpainsymman.2017.03.008
        • Wright AA
        • Zhang B
        • Ray A
        • et al.
        Associations between end-of-life discussions, patient mental health, medical care near death, and caregiver bereavement adjustment.
        JAMA. 2008; 300 (PMID: 18840840; PMCID: PMC2853806): 1665-1673https://doi.org/10.1001/jama.300.14.1665
        • Miyajima K
        • Fujisawa D
        • Yoshimura K
        • et al.
        Association between quality of end-of-life care and possible complicated grief among bereaved family members.
        J Palliat Med. 2014; 17 (Epub 2014 Jul 22. PMID: 25050607): 1025-1031https://doi.org/10.1089/jpm.2013.0552
      1. The Economist Intelligence Unit. The 2015 quality of death index. Lien Foundation. Available at:https://www.eiuperspectives.economist.com/sites/default/files/2015%20EIU%20Quality%20of%20Death%20Index%20Oct%2029%20FINAL.pdf. Accessed November 16, 2021.

        • Mayland CR
        • Williams EM
        • Ellershaw JE.
        Assessing quality of care for the dying: the development and initial validation of a postal self-completion questionnaire for bereaved relatives.
        Palliat Med. 2012; 26 (Epub 2011 Nov 1. PMID: 22045726): 897-907https://doi.org/10.1177/0269216311424953
        • Mayland CR
        • Williams EM
        • Addington-Hall J
        • et al.
        Does the ‘Liverpool Care Pathway’ facilitate an improvement in quality of care for dying cancer patients?.
        Br J Cancer. 2013; 108: 1942-1948https://doi.org/10.1038/bjc.2013.203
        • Mayland CR
        • Williams EM
        • Addington-Hall J
        • et al.
        Assessing the quality of care for dying patients from the bereaved relatives' perspective: further validation of “Evaluating care and health outcomes–for the dying”.
        J Pain Symptom Manage. 2014; 47 (Epub 2013 Nov 5. PMID: 24210451): 687-696https://doi.org/10.1016/j.jpainsymman.2013.05.013
        • Mayland C
        • Williams E
        • Ellershaw J.
        How well do current instruments using bereaved relatives' views evaluate care for dying patients?.
        Palliat Med. 2008; 22 (PMID: 18372378): 133-144https://doi.org/10.1177/0269216307085742
        • World Health Organization
        Palliative care.
        WHO, 2020 (Available at:) (Accessed August 21, 2021)
        • Mayland CR
        • Lees C
        • Germain A
        • et al.
        Caring for those who die at home: the use and validation of ‘Care Of the Dying Evaluation’ (CODE) with bereaved relatives.
        BMJ Support Palliat Care. 2014; 4 (Epub 2014 Mar 28. PMID: 24681559): 167-174https://doi.org/10.1136/bmjspcare-2013-000596
        • Mayland CR
        • Mulholland H
        • Gambles M
        • et al.
        How well do we currently care for our dying patients in acute hospitals: the views of the bereaved relatives?.
        BMJ Supportive & Palliative Care. 2017; 7: 316-325
        • Vogt A
        • Stiel S
        • Heckel M
        • et al.
        Assessment of the quality of end-of-life care: translation and validation of the German version of the “Care of the Dying Evaluation” (CODE-GER) - a questionnaire for bereaved relatives.
        Health Qual Life Outcomes. 2020; 18 (PMID: 32962706; PMCID: PMC7507719): 311https://doi.org/10.1186/s12955-020-01473-2
        • Mayland C
        • McGlinchey T
        • Gambles M
        • et al.
        Quality assurance for care of the dying: engaging with clinical services to facilitate a regional cross-sectional survey of bereaved relatives' views.
        BMC Health Serv Res. 2018; 18 (PMID: 30305082; PMCID: PMC6180653): 761https://doi.org/10.1186/s12913-018-3558-z
        • Kupeli N
        • Candy B
        • Tamura-Rose G
        • et al.
        Tools measuring quality of death, dying, and care, completed after death: systematic review of psychometric properties.
        Patient. 2019; 12 (PMID: 30141020; PMCID: PMC6397142): 183-197https://doi.org/10.1007/s40271-018-0328-2
        • Haugen DF
        • Hufthammer KO
        • Gerlach C
        • et al.
        ERANet-LAC CODE Project Group. good quality care for cancer patients dying in hospitals, but information needs unmet: bereaved relatives' survey within seven countries.
        Oncologist. 2021; 26 (Epub 2021 Jun 17. PMID: 34060705; PMCID: PMC8265351): e1273-e1284https://doi.org/10.1002/onco.13837
        • Mayland CR
        • Gerlach C
        • Sigurdardottir K
        • et al.
        Assessing quality of care for the dying from the bereaved relatives' perspective: Using pre-testing survey methods across seven countries to develop an international outcome measure.
        Palliat Med. 2019; 33 (Epub 2019 Jan 10. PMID: 30628867; PMCID: PMC6376606): 357-368https://doi.org/10.1177/0269216318818299
        • Jones J
        • Hunter D.
        Consensus methods for medical and health services research.
        BMJ. 1995; 311: 376-380
      2. National Health Service. Friends and family test. Available at: https://www.england.nhs.uk/fft. Accessed September 25, 2021.

        • Flora DB
        • Curran PJ.
        An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data.
        Psychol Methods. 2004; 9 (PMID: 15598100; PMCID: PMC3153362): 466-491https://doi.org/10.1037/1082-989X.9.4.466
        • Muthen LK
        • Muthen BO.
        Mplus Users Guide (Sixth ed.).
        Muthen & Muthen, Los Angeles, CA1998 (-2011)
        • MacCallum RC
        • Browne MW
        • Sugawara HM.
        Power analysis and determination of sample size for covariance structure modeling.
        Psychological Methods. 1996; 1: 130-149https://doi.org/10.1037/1082-989X.1.2.130
        • Hu L-t
        • Bentler PM.
        Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives.
        Structural Equation Modeling. 1999; 6: 1-55https://doi.org/10.1080/10705519909540118
        • Schreiber JB
        • Stage FK
        • King J
        • et al.
        Reporting structural equation modeling and confirmatory factor analysis results: a review.
        The Journal of Educational Research. 2006; 99: 323-337https://doi.org/10.3200/JOER.99.6.323-338
        • Watt T
        • Groenvold M
        • Deng N
        • et al.
        Confirmatory factor analysis of the thyroid-related quality of life questionnaire ThyPRO.
        Health Qual Life Outcomes. 2014; 12 (PMID: 25213857; PMCID: PMC4172819): 126https://doi.org/10.1186/s12955-014-0126-z
        • Fayers PM
        • Machin D.
        Quality of life: the assessment, analysis and interpretation of patient-reported outcomes.
        John Wiley & Sons, Chichester2013
        • Cohen J.
        Statistical power analysis for the behavior science.
        Lawrance Eribaum Association, New York1988
        • Hanson BA.
        Uniform DIF and DIF Defined by Differences in Item Response Functions.
        Journal of Educational and Behavioral Statistics. 1998; 23: 244-253https://doi.org/10.3102/10769986023003244
        • Zumbo BD.
        A handbook on the theory and methods of differential item functioning (DIF).
        National Defense Headquarters, Ottawa1999
        • Nagelkerke NJ.
        A note on a general definition of the coefficient of determination.
        Biometrika. 1991; 78: 691-692
        • Reise S.P.
        • Moore T.M.
        • Haviland M.G.
        Bifactor models and rotations: Exploring the extent to which multidimensional data yield univocal scale scores.
        Journal of Personality Assessment. 2010; 92: 544-559
        • Reise SP
        • Bonifay WE
        • Haviland MG.
        Scoring and modeling psychological measures in the presence of multidimensionality.
        J Pers Assess. 2013; 95 (Epub 2012 Oct 2. PMID: 23030794): 129-140https://doi.org/10.1080/00223891.2012.725437
        • Keetharuth AD
        • Bjorner JB
        • Barkham M
        • et al.
        Exploring the item sets of the Recovering Quality of Life (ReQoL) measures using factor analysis.
        Qual Life Res. 2019; 28 (Epub 2018 Dec 21. PMID: 30578454; PMCID: PMC6439178): 1005-1015https://doi.org/10.1007/s11136-018-2091-1
        • Rubinstein A
        • Zerbino MC
        • Cejas C
        • et al.
        Making universal health care effective in argentina: a blueprint for reform.
        Health Syst Reform. 2018; 4: 203-213https://doi.org/10.1080/23288604.2018.1477537
        • Goldraij G
        • Tripodoro VA
        • Aloisio M
        One chance to get it right: improving clinical handovers for better symptom control at the end of life.
        BMJ Open Quality. 2021; 10e001436https://doi.org/10.1136/bmjoq-2021-001436
      3. Pérez-Cruz PE, Padilla Pérez O, Bonati P, et al. Validation of the Spanish Version of the Quality of Dying and Death Questionnaire (QODD-ESP) in a Home-Based Cancer Palliative Care Program and Development of the QODD-ESP-12. J Pain Symptom Manage. 2017;53:1042-1049 .e3.https://doi.org/10.1016/j.jpainsymman.2017.02.005. Epub 2017 Mar 16. PMID: 28323080.