Special Article| Volume 63, ISSUE 4, e419-e429, April 2022

Ok

# Cross Country Comparison of Expert Assessments of the Quality of Death and Dying 2021

Open AccessPublished:December 21, 2021

## Abstract

### Context

Few efforts have attempted to quantify how well countries deliver end-of-life (EOL) care.

### Objectives

To score, grade, and rank countries (and Hong Kong and Taiwan) on the quality of EOL care based on assessments from country experts using a novel preference-based scoring algorithm.

### Methods

We fielded a survey to country experts around the world, asking them to assess the performance of their country on 13 key indicators of EOL care. Results were combined with preference weights from caregiver-proxies of recently deceased patients to generate a preference-weighted summary score. The scores were then converted to grades (from A–F) and a ranking was created for all included countries.

### Results

The final sample included responses from 181 experts representing 81 countries with 2 or more experts reporting. The 6 countries who received the highest assessment scores and a grade of A were United Kingdom, Ireland, Taiwan, Australia, Republic of Korea, and Costa Rica. Only Costa Rica (upper middle) is not a high income country. Not until Uganda (ranked 31st) does a low-income country appear on the ranking. Based on the assessment scores, twenty-one countries received a failing grade, with only two – Czech Republic (66th), and Portugal (75th) – being high income countries.

### Conclusion

This study provides an example of how a preference-based scoring algorithm and input from key stakeholders can be used to assess EOL health system performance. Results highlight the large disparities in assessments of the quality of EOL care across countries, and especially between the highest income countries and others.

## Key Message

Based on input from country experts, nearly half of countries received a grade of D or F in the quality of death, and dying. To improve, countries should model the enabling factors observed in the high performing countries.

## Introduction

Over 55 million people died in 2019.
World Health Organisation
Global health estimates – Estimated deaths by cause, sex, and WHO member state.
The vast majority died in low and middle-income countries (LMICs) without even basic access to palliative care, an essential component of universal health coverage. As a result, many of these individuals died in pain, and with significant distress. Yet, living in a country with, or even receiving, specialized palliative care does not guarantee a high-quality end-of-life experience. Despite a lack of systematic reporting in most countries, there is overwhelming evidence that even high-functioning health systems often fail to deliver on aspects of end-of-life care that are important to patients with advanced illness and their caregivers. Most end-of-life (EOL) patients, even in high income countries, die in pain and after experiencing serious health-related suffering, not at their place of choice, under significant psychological distress, and, prior to death, often express regret about how the last period of their life was spent.
• Knaul FM
• Farmer PE
• Krakauer EL
• et al.
Alleviating the access abyss in palliative care and pain relief-an imperative of universal health coverage: the Lancet Commission report.
• Singer AE
• Goebel JR
• Kim YS
• et al.
Populations and interventions for palliative and end-of-life care: a systematic review.
• Clark D
• Baur N
• Clelland D
• et al.
Mapping levels of palliative care development in 198 countries: the situation in 2017.
Many patients also routinely undergo expensive and marginal or non-beneficial treatments that may inadvertently worsen their EOL experience.
• Cardona-Morrell M
• Kim JCH
• Brabrand M
• Gallego-Luxan B
• Hillman K.
What is inappropriate hospital use for elderly people near the end of life? A systematic review.
As a result, poor care, serious health-related suffering, medical bankruptcy and treatment regret are common in both more and less developed countries.
Despite extensive efforts to measure the quality of curative treatments across countries, few efforts have focused on cross-country comparisons of EOL care. Those that have been conducted applied the Donabedian framework
• Donabedian A.
Evaluating the quality of medical care.
and largely focused on measuring structural inputs (e.g., funding, policies, personnel, education) and/or proxy outcomes as indicators for EOL care. While important components of care delivery, these studies make the assumption that countries that score higher on such measures necessarily deliver a better EOL experience.
• Stjernswärd J
• Foley KM
• Ferris FD.
Integrating palliative care into national policies.
For example, in 2015, the Economist Intelligence Unit (EIU) published the second iteration of the Quality of Death Index
The Economists Intelligence Unit
The 2015 quality of death index: ranking palliative care across the world.
(QODI) which ranked the quality of palliative care in 80 countries. This index included a combination of both inputs and health outcomes likely to be correlated with the quality of palliative care delivery using 20 indicators across five categories with weighting criteria as follows:
• Palliative and healthcare environment (20% weighting; 4 indicators)
• Human resources (20% weighting; 5 indicators)
• Affordability of care (20% weighting; 3 indicators)
• Quality of Care (30% weighting; 6 indicators)
• Community engagement (10%; 2 indicators)
Indicators were both quantitative and qualitative and included such measures as median life expectancy, healthcare spending as a percentage of gross domestic product (GDP), the existence of public awareness campaigns focusing on end-of-life care, and whether or not the country had a government-led national palliative care strategy, among others.
Although useful for drawing attention to the low quality of EOL care in many countries, the prior QODIs, and similar efforts have significant limitations. For curative treatment, it can be expected that countries largely focus on preventing disability and premature mortality, followed by cost and equity concerns, which is why these indicators are commonly tracked in countries worldwide. Yet, for dying patients and their family caregivers, there are many other considerations that may be equally or more important.
• Steinhauser KE
• Clipp EC
• McNeilly M
• Christakis NA
• McIntyre LM
• Tulsky JA.
In search of a good death: observations of patients, families, and providers.
For example, dying at place of choice and having access to friends and family, a significant concern in COVID-19 care, could matter more than marginal increases in life extension or even pain management.
• Finkelstein EA
• Bilger M
• Flynn TN
• Malhotra C.
Preferences for end-of-life care among community-dwelling older adults and patients with advanced cancer: a discrete choice experiment.
Incorporating the relative value of what matters most to patients and families at EOL is tantamount for assessing the quality of EOL care delivery. However prior efforts had only limited input from patients and families. Second, in efforts to standardize across countries, prior indicators were chosen largely based on data availability. It is unclear how well many of these indicators correlate with actual EOL care delivery.
To overcome these limitations, in this effort we take an alternative approach. We systematically rank and grade countries on the quality of EOL care taking patient and caregiver preferences into account but relying on input from experts to provide their assessment of EOL performance on key indicators. This low-cost approach, which has been used in ranking efforts beyond palliative care, can be a complement or substitute to traditional data driven approaches and can be applied both within and across countries.
• Olmsted MG
• Geisen G
• Murphy J
• Bell D
• Morley M
• Stanley M
Methodology: U.S. News & World Report Best Hospitals 2015-16.

## Methods

This effort comprised three stages. First, we undertook an extensive scoping review that identified the core domains of end-of-life care that matter most to patients, and caregivers (Bhadelia, A et al, 2021 forthcoming).
• Oldfield LE
• Cruz JL
• Singh R
• Finkelstein EA.
Identifying the core domains to assess the ‘quality of death’: a scoping review.
Second, we developed a discrete choice experiment (DCE) survey to produce preference weights associated with each of the key indicators of EOL care identified in the scoping review (Gonzalez-Sepulveda, JM 2021 forthcoming).
• Gonzalez-Sepulveda JM
• Baid D
• Johnson FR
• Finkelstein EA.
What is a good death? A choice experiment on care indicators for patients at end of life.
Finally, as reported in this article, we developed, and fielded a survey for country-experts around the world who were qualified to assess the quality of EOL care within their country. A preference weighted summary score was computed for each country by combining the responses from the country-experts with the preference weights for each response option. Using the summary scores, countries were assigned a ranking, and grade from A to F.
This manuscript begins with a summary of the methods and results of the scoping review and DCE exercise, which were the two foundational efforts required to produce the assessment scores. We then present the methods for generating the final ranking and grades. This is followed by a reporting of the results, including a qualitative analysis of open-ended questions that allowed the country experts to report critical factors that underlie their ratings for the key indicators. We conclude with a summary and discussion of what countries can do to improve EOL care. We also include a discussion of limitations of this effort and a research agenda for how it can be improved in future iterations.

### Identifying Key Indicators for EOL Care

The scoping review that formed the basis of the indicators used in this effort systematically analyzed 309 articles to identify a comprehensive set of domains and subdomains considered to be important for improving the ‘quality of death and dying’ by patients and/or caregivers. The review focused on systems level inputs (e.g., access, stewardship, governance) and patient and caregiver level outcomes. Outcomes focused on quality indicators, including quality of care, quality of communication, and affordability.

### Estimating Preference-Weights for Key Indicators

Using the results of the review, with input from an external Advisory Board, and cognitive interviews, Gonzalez-Sepulveda et al derived a set of 13 indicators that can be used to assess the quality of EOL care delivery. The indicators are listed below. Each indicator included five response options on a Likert scale ranging from strongly disagree to strongly agree:
• 1.
The places where health care providers treated patient were clean, safe, and comfortable
• 2.
Patient was able to be cared for and die at place of choice
• 3.
Health care providers provided appropriate levels and quality of life extending treatments
• 4.
Health care professionals supported patient's spiritual, religious, and cultural needs
• 5.
Care was well coordinated across different healthcare providers
• 6.
Health care providers controlled pain and discomfort to patient's desired levels
• 7.
Health care providers helped patient cope emotionally
• 8.
Health care providers encouraged contact with friends and family
• 9.
Health care providers helped with patient's non-medical concerns
• 10.
Health care providers delivered clear and timely information so patients could make informed decisions
• 11.
Health care providers asked enough questions to understand patient's needs
• 12.
Health care providers mostly treated patients kindly and sympathetically
• 13.
Costs were not a barrier to patient getting appropriate care
The authors then fielded a discrete choice experiment (DCE) survey to 1250 family caregivers of deceased EOL patients across five countries to quantify the relative importance of each indicator and the value of moving from lower to higher levels within indicators. The results not only showed preference variation across the 13 indicators, as reproduced in Fig. 1 below, but also that respondents placed greater value on improvements from very low to low levels (i.e., from strongly disagree to disagree) than from high to very high levels (i.e., agree to strongly agree). The preference weights for each level of each indicator are reproduced in Appendix Table A.

### Survey Development

The DCE exercise provided preference weights for each response option. To generate the assessment scores, we developed a survey instrument that included the 13 statements shown above along with detailed explanations and the five Likert response options and fielded it to country experts around the world. The experts were asked to respond based on their views of expected outcomes for patients with life limiting illnesses in their country; they also had the option to respond with “Don't know” or “Not applicable.”
In addition to the 13 indicator questions, the survey also included four Yes/No questions concerning 1) whether palliative is included in a national law on healthcare in the country, 2) whether the country has at least one government-backed written strategy or plan at the national level for the delivery of palliative care, 3) whether there is a person, desk or unit in the national government responsible for palliative care, and 4) whether palliative care is a recognized specialty or sub-speciality in the country. A fifth question focused on availability of opioids for pain relief for palliative care patients throughout the country with six response options ranging from 1 (not available) to 6 (all opioids are always available). These questions were included to gauge the reliability of the results. The expectation is that there should be a positive correlation between each country's responses to these questions and their QODDI assessment score.
The survey also included the following open-ended questions:
• How would you rate your country on the delivery of end-of-life care services relative to countries with similar levels of economic development?
• What are the key factors that your country does well in terms of the delivery of end-of-life care services that led you to choose this rating?
• What are the key factors that your country does poorly in terms of the delivery of end-of-life care services that led you to choose this rating?
• What do you see as the most important action your country could take to improve the quality of end-of-life care services in your country?
Responses to these questions provide context to the response options chosen by respondents. Other sections, including two discrete choice experiments, patient vignettes, and quality of care provided to caregivers of EOL patients will be the subject of future manuscripts. The full survey instrument is available at Supplementary document E.

### Study Setting and Participants

To select the countries for inclusion in the index, we started with all 217 countries and territories identified by the World Bank and included Taiwan. We excluded 55 countries with less than 2 million population and North Korea and Congo due to concerns it would be cost-prohibitive to identify survey respondents from these countries. This left 161 countries in our sampling frame. A further 80 countries are not included in the final ranking because less than two country experts responded to our survey request (30 countries with only one response and 50 countries with zero responses). This left 81 countries to be included for the final ranking exercise; these countries comprise 81% of the world's population.

Bank W. World Data. The World Bank. Accessed from: https://data.worldbank.org/region/world

We focused the ranking survey on country experts who are qualified to respond on the general levels of EOL care delivery in the country as a whole. To be eligible, experts had to be either 1) a representative of the national in-country hospice-palliative care association or similar national professional association (e.g., hospice council) with an established leadership role, 2) a health care provider (physician, nurse) involved in provision of palliative care 3) a government employee or academic with knowledge of palliative care in the country, and 4) at least 21 years of age, and 5) able to communicate in English. We aimed to recruit at least two such experts within each country.
Experts were identified with input from an external Advisory Board of palliative care experts worldwide and through regional contacts. Initially, 414 experts were invited to take part in the survey. Each KI was emailed an individualized link through which they could complete the survey online. Participants were briefed on the objectives of the study, privacy and confidentiality of data and given the option to be acknowledged for their inputs in the final report. Reminders were sent on a weekly basis. If after three reminders no response was obtained and two respondents had not yet completed the survey from that country, we identified, and emailed a survey link to another KI. Although we aimed for two responses per country, as some KIs responded after two experts were identified, we included their data when generating the results. The study was reviewed and approved by the National University of Singapore Institutional Review Board (protocol reference code: 2020–218).

### Data Analysis

To score countries on EOL care delivery, we first quantified a total score for each respondent for each of the 13 indicators and then averaged across respondents within each country. Prior to calculating the respondent score, responses of “Don't know” or “Not applicable”, which occurred in less than 1% of the total, were imputed with the mean value of the respondent's non-missing scores. For a given expert k, the overall score is given by the sum of the preference weight associated with indicator j, and quality-rating level i:
$Vk=∑j=113wijk$

All scores were rescaled to between 0 and 100 using the following formula:
$Overallscorek=Vk−VminVmax−Vmin×100$

$Vmax=∑j=113w5j=maximumscoreobtainedifeveryindicatorhada5−starqualityrating$

$Vmin=∑j=113w1j=minimumscoreobtainedifeveryindicatorhada1−starqualityrating$

We then averaged the rescaled scores across experts within each country to obtain an overall score for the country. We then assigned a grade (A best–F worst) for each score using a ten-point scale (i.e., 90–100 = A, 80–90 = B…). The grades provide additional insight as to how well each country is delivering EOL care based on the KI assessments.

### Validity Checks

In efforts to gauge the validity of our approach we present the correlation between the assessment scores and the 2020 Human Development Index,
Programme UND
The 2020 human development report.
2019 per capita GDP,

Bank W. GDP per capita, PPP (current international $). The World Bank. Accessed from: https://data.worldbank.org/indicator/NY.GDP.PCAP.PP.CD 2017 Level of Palliative Care Development, • Clark D • Baur N • Clelland D • et al. Mapping levels of palliative care development in 198 countries: the situation in 2017. 2015 EIU scores, The Economists Intelligence Unit The 2015 quality of death index: ranking palliative care across the world. and responses to the five survey questions focusing on key inputs thought to correlate with high quality EOL care. We expect a positive correlation between QODDI assessment scores and each of these metrics. Lastly, given concerns that our scores are based on inputs from only a few experts per country, we report within country variability across respondents using the following formula: $Inter−rateragreement=1−∑i=113(x1i−x2i)2∑i=113(xmax−xmin)2$ where $x1i$ and $x2i$ are rater 1 and rater 2’s responses to attribute $i$ respectively; $xmax=5$ and $xmin=1$, correspond to “Strongly Agree” and “Strongly Disagree” responses respectively. Agreement scores range from 0 (completely divergent responses), to 1 (total agreement). When there are more than two experts per country, pair-wise inter-rater scores were calculated and averaged to generate a score for the country. Low agreement scores cast doubt on the validity of our approach. ## Results ### Sample Characteristics Of the 414 country experts invited to respond to the survey between May and August 2021; 236 responded, for a response rate of 57%. However, of these, responses from 55 experts were excluded as, despite our best efforts, we could not identify a second respondent from their country. Best efforts included seeking help for additional names from identified country experts, regional Palliative Care Association members, and our Advisory Board members. All names provided were approach via an introductory email message and two subsequent reminders. The final analyses included responses from 181 experts representing 81 countries. These countries represent only 37% of all countries but encompass 81% of the world's population. Table 1 shows the distribution of countries included in the index by geographic region and World Bank income groups. As can be seen from the table, 69% of countries were either high or upper-middle income countries, and only 22% are from the African continent. Appendix Table B shows similar results for countries where two respondents could not be identified. Excluded countries were far more likely to be lower middle/low income countries and from Middle East & North Africa or Sub-Saharan Africa. Table 1Geographic and Income Group Distribution of Included Countries Geographic Region/ Income LevelHigh IncomeUpper Middle IncomeLower Middle IncomeLow IncomeTotal East Asia & Pacific74415 Europe & Central Asia195226 Latin America & Caribbean3102116 Middle East & North Africa1315 North America22 South Asia44 Sub-Saharan Africa27413 Total322420581 ### Unweighted Indicator Ratings From Country-Experts Fig. 2 presents the unweighted responses for each of the 13 indicators for the 181 respondents across the 81 countries. Several insights can be gleaned from the Fig. First, there is substantial variation across response options for each of the indicators. However, four (agree), followed by two (disagree), were the most common responses for most indicators. Moreover, 9 of the 13 indicators follow a similar, and slightly right skewed distribution. This suggests a high correlation of responses across indicators for a given country but also that assessments of EOL care tends to be more favorable than unfavorable. Respondents were least likely to respond with ‘strongly disagree’ to any of the indicator statements. For only 3 of the 13 indicators (coordination of care, non-medical concerns, and preferred place of death) were there more ‘strongly disagree’ than ‘strongly agree’ responses. Fig. 3 presents a bar chart of the rankings of the 81 countries. Appendix Table C supplements this information with the rank, score, and grade for each country, along with the income group, geographic region, and number of country experts responding. Rounding out the top 6 countries, those who received a grade of A, were United Kingdom, Ireland, Taiwan, Australia, Republic of Korea, and Costa Rica. Only Costa Rica (Upper Middle) is not a High income country. Not until Uganda appears at #31 does a low income country appear on the ranking. At the other end of the range, 21 countries received a failing grade based on the assessments. Of these, only two, Czech Republic at #66 and Portugal at #75, are high income countries and 11 are either Low or Lower income countries. The highest-ranking country (ranked 27th) from Africa is Botswana, an upper middle-income country. ### Face Validity Tests Tests reveal that the correlation between the QODDI scores and the 2020 Human Development Index and 2019 per capita GDP is 0.53 and 0.55 respectively. These are moderately high correlations which provides some evidence that the results are plausible given the high correlations between income and health system performance in general. Stratifying QODDI scores by income group, as shown in Fig. 4, further elucidates the relationship between the assessment scores, and wealth at the country level. The fig. reveals that the highest income countries score best but differences between the remaining three countries are not statistically significant. Additional details of this relationship are shown Fig. D in the Appendix. This fig. presents the average raw scores for each of the 13 indicators by income group. For all but one indicator, spiritual needs, high income countries scored the highest. For several indicators, including cleanliness, emotional coping, costs not being barriers to care, and timeliness, there is a clear ordering from worst to best by income group, providing further evidence of the validity of the approach. Perhaps not surprisingly, the largest gradient occurs for the affordability indicator. However, for many indicators, low-income countries did nearly as well, or in some cases better, than lower- or upper middle-income countries. These results provide evidence that financial constraints are not the only barrier to high quality EOL care. To this point, our index showed a correlation of 0.35 with the countries’ universal healthcare coverage status. We found even stronger correlations of 0.58 and 0.61 between our index and the 2015 EIU scores The Economists Intelligence Unit The 2015 quality of death index: ranking palliative care across the world. and the 2017 Palliative Care Development Index, • Clark D • Baur N • Clelland D • et al. Mapping levels of palliative care development in 198 countries: the situation in 2017. both of which include measures of a country's wealth, and access to care but also other measures assumed to be correlated with high quality EOL care. We further found that our index showed a moderate correlation of 0.48 with the survey question reporting the country's availability of opioids for pain relief for palliative care patients. However, this is partly because country's that score highly on pain relief also score highly on other inputs that are expected to improve the EOL experience of patients. We also found positive correlations between our index with whether there is a person, desk or unit in the national government responsible for palliative care and/or hospice (r = 0.39), whether there is at least one government-backed written strategy or plan at the national level for the delivery of palliative care (r = 0.30), and whether palliative care is included in a national law on health care (r = 0.22). Whether palliative care is a recognized healthcare specialty or sub-specialty within the country showed only a small correlation (r = 0.09) with the QODDI scores. Finally, within-country agreement scores for the 81 included countries averaged 0.86, suggesting a high degree of agreement across experts within countries. Factors that experts identified as critical to high (low) quality EOL care can be seen in Panel 1. Panel 1: Country Expert Responses to Open-ended Survey Questions Country experts reported enabling and disabling factors that influenced their countries progress toward achieving high-quality EOL care. Factors that entered positively included: • 1) Uninterrupted and adequate access to opioids and other essential medicines to alleviate serious health-related suffering at the EoL. • Knaul FM • Farmer PE • Krakauer EL • et al. Alleviating the access abyss in palliative care and pain relief-an imperative of universal health coverage: the Lancet Commission report. • 2) Evidence-based and equity-oriented policies on and investments in palliative care at the national level. • 3) Personalized, patient-centered, and integrated care delivery. • 4) Legislated entitlement to publicly financed universal health coverage that ensures free or low-cost access to palliative and EOL services, including community, and home-based services. • 5) Institutionalized regulatory mechanisms and government oversight to ensure quality standards. • 6) Mandatory competency-based palliative care education and training for clinicians and allied health professionals to generate highly skilled multidisciplinary healthcare teams qualified to manage patients with life limiting illnesses. • 7) Intersectional and multisectoral approaches that bridge boundaries between public and private entities, and social and health services. • 8) Public education to promote awareness on and support civil society engagement on topics related to EOL. • 9) Promotion of compassionate communities that complement formal EoL care services to improve quality of life of patients and their families. This includes training and support for lay caregivers. • 10) Research opportunities and activities to generate necessary evidence for palliative care development. Factors identified as contributing negatively to EOL care include: • 1) Lack of a recognition that EOL care is a human right • 2) Absence of a national strategy for palliative care/not making palliative care a priority at the national level. • 3) Limited or no integration of palliative care into the broader health system, • 4) Lack of investment in EOL care leading to human resource shortages (i.e., lack of palliative care specialists), shortfalls in access to opioids, lack of dedicated facilities for patients at EOL, and no or limited access to home-based palliative and psychosocial care, • 5) Lack of generalist or specialist palliative care training. • 6) High cost of EOL care. • 7) Lack of public knowledge and awareness of availability palliative care services • 8) Poor or limited communication with patients and families on EOL care options • 9) Lack of patient autonomy to make EoL decisions. • 10) Lack of recognition of cultural factors associated with death and dying ## Discussion Peter Drucker, one of the world's great management consultants, once stated “if you can't measure it, you can't improve it.” This holds true for businesses and governments alike. Yet, when it comes to end-of-life care, too few efforts have been undertaken to measure, and improve performance. This is why the two prior QODI indices and similar efforts are so important. Despite their limitations, they did what many governments are hesitant to do; they measured and held governments accountable for their, too often poor, performance. And in fact, many governments responded by making efforts to improve their performance, or at least their ranking. Singapore, for example, embarked on an exercise to develop a National Strategy for Palliative Care in direct response to their position in the initial QODI. Our goal in producing the country expert assessment scores, grads, and ranking is in the same spirit as the prior efforts. We want to measure and hold countries accountable for their performance in efforts to improve the EOL experience for patients and families. We recognize that measuring quality of end-of-life care is an important and evolving process. Concerns that one could improve the rankings using the algorithms employed in the prior approaches but without necessarily improving what matters most to patients and families at EOL led us to take a novel approach to measuring quality. Our approach overcomes three major shortcomings of the prior efforts. First, by focusing on outcomes, as opposed to inputs, we do not have concerns that countries could improve their ranking without necessarily improving EOL care delivery. Second, our approach is not limited by data availability, and data quality, which are highly variable across countries. This forced prior efforts to focus on a limited set of proxy measures, some of which are only loosely correlated with EOL outcomes. Third, and most importantly, we take care to weight our indicators, and levels within indicators, based on what matters most to patients and caregivers at EOL. This approach provides a clearer link between the assessment scores and overall quality of EOL care delivery. It also recognizes that the value that patients place on EOL care is non-linear. With limited resources, in addition to focusing on the indicators that are weighted highest, policymakers should work to avoid very low ratings rather than place additional resources on achieving the highest ratings. This results because there is diminishing marginal value associated with increasingly higher ratings. This is not to say that our approach is not without limitations. First, our approach is limited to 13 indicators. Although based on the best available evidence, there may be other indicators that are important to patients at EOL that should be included. For example, we considered trying to incorporate the ability of providers to help patients with life limiting illnesses to maintain hope as an additional indicator. However, after several rounds of cognitive interviews, we abandoned this effort as we struggled to operationalize hope with a single indicator, and our approach limits the number of indicators that can be included. We also did not include indicators for family caregivers, such as a bereavement indicator, although high quality EOL care should focus on both patients, and caregivers. Although our approach to weighing the indicators is a unique contribution, ideally the weights would come from patients themselves. Because of cost, time, and practical constraints, we relied on family caregivers of recently deceased patients as proxies. We also used weights from an average across five countries; inclusion of additional countries could change the weights. Each country, or for that matter each EOL patient, likely has their own weights, and efforts to maximize EOL care delivery should ideally take a more granular approach. This would allow for care delivery that is both high quality and culturally appropriate. In a perfect world, assessment scores would come from nationally representative samples of patients at EOL, or their proxies, from countries around the world, as they are in the best position to assess quality of EOL care delivery. However, that was not possible given time, and resource constraints. As an admittedly inferior alternative, we relied on inputs from experts knowledgeable to respond on behalf of the country as a whole and selected based on consistent criteria applied across countries in efforts to minimize selection bias. Despite our best efforts, and perhaps partly a result of the high clinical demands on palliative care providers due to Covid-19, lack of responses limited the analysis to only a few experts per country for 81 countries. We could not identify two respondents for an additional 80 countries who we suspect would perform poorly on the index. This results both because lack of identifiable experts is a likely signal of low quality and because these countries are more likely to be lower income, which our results show is also a signal for low scores. For countries that are included, although within-country agreement scores averaged 0.86, suggesting a high degree of agreement, the potential for bias remains. This bias may be exacerbated for larger countries where experts may have less knowledge about the state of palliative care in other regions. Future efforts that rely on input from experts should aim to include as large a representative sample as possible in efforts to minimize potential biases and produce more reliable estimates. As with all prior ranking efforts, we have ignored variability. Many of the differences between countries are very close and not statistically significant. For this reason, it may be more informative to focus on the country's grade, as opposed to ranking, as an overall indicator of where the country stands in terms of the quality of EOL care delivery based on the country expert assessments. Lastly, the current endeavor does not capture inequities in EOL care within countries, specifically among disadvantaged communities. All of these limitations can and should be improved in future efforts. Despite these limitations, our results, including the ranking and grade, provide an additional indicator to allow policymakers to take stock of where they are in terms of EOL care delivery as perceived by within-country experts. Policymakers should consider these assessment scores, along with other measures of EOL care delivery, to determine whether further actions are needed. Panel 1 provides clear direction for what countries can do to directly improve EOL care. This includes increasing access to opioids and other essential medicines to relieve serious health-related suffering, universal health coverage of EoL services to avoid medical bankruptcy, education and training programs to increase capacity to deliver high quality EOL care (including by lay health workers), integrated care delivery, and greater oversight to increase quality, including quality of communication. Panel 1 also points to factors that can indirectly improve EOL care delivery, including public education programs, promotion of compassionate communities, and higher investments in research. These recommendations are not novel and echo those in recent related efforts, • Knaul FM • Farmer PE • Krakauer EL • et al. Alleviating the access abyss in palliative care and pain relief-an imperative of universal health coverage: the Lancet Commission report. , World Health Organisation. Assessing the development of palliative care worldwide: a set of actionable indicators. World Health Organization. Accessed from: https://www.who.int/publications/i/item/9789240033351 , • Connor SR • Centeno C • Garralda E • Clelland D • Clark D. Estimating the number of patients receiving specialized palliative care globally in 2017. but with the renewed attention to death, and dying as a result of Covid-19, perhaps the time is right to act on these recommendations. This is especially true for efforts to manage pain and discomfort given this was the highest ranking indicator and the fact that many patients still die without access to even basic pain medications. The results also show that efforts to improve communication and the quality of the environment where care is delivered can greatly improve the EOL experience, especially in places where baseline levels are low. These improvements can potentially be done with only minimal investment and should be strongly considered. ## Conclusion This study provides an application for how preference-weighted assessment scores can be used to quantify EOL care delivery across countries. The results highlight the great disparities in the quality of EOL care across countries and especially between the highest income countries and others. The finding that, based on assessment scores provided by country experts, nearly half of countries received a grade of D or F in the quality of death and dying should serve as a stark reminder of the lack of attention to EOL care and be a call to action to improve performance. Enabling factors observed in many of the high performing countries provide a clear strategy and starting point to improve the EOL experience for patients and families world-wide. Notably, greater integration of palliative care into the community and within the broader health system, can have a far-reaching impact on ensuring high-quality EOL care for all, particularly from an equity perspective. This effort is intended to provide grounding evidence and build awareness – social, political, and financial – of EOL care as an important public health and societal issue. It seeks to motivate policymakers, providers, community stakeholders and the general public to action to improve EOL performance through advocacy, policy change, resource mobilization, implementation, and ongoing monitoring and evaluation. ### Funding This study was funded by the Lien Foundation. The funder had no role in the study design, data collection, analysis, and decision to publish or preparation of the manuscript. ### Disclosures and Acknowledgments We would like to thank the external Advisory Board members for their contribution to various study related discussions and feedback during survey development and Vinh Anh Huynh, research staff at the Lien Centre for Palliative Care for supporting data analyses. We also thank all the country experts who took the time to complete our survey thus making this effort possible. ## Appendix. Supplementary materials • Supplementary materials Available online and upon request from the authors. ## References • World Health Organisation Global health estimates – Estimated deaths by cause, sex, and WHO member state. World Health Organization, 2019 (Accessed from:) ((Accessed December 1, 2021) • Knaul FM • Farmer PE • Krakauer EL • et al. Alleviating the access abyss in palliative care and pain relief-an imperative of universal health coverage: the Lancet Commission report. Lancet. 2018; 391: 1391-1454https://doi.org/10.1016/s0140-6736(17)32513-8 • Singer AE • Goebel JR • Kim YS • et al. Populations and interventions for palliative and end-of-life care: a systematic review. J Palliat Med. 2016; 19: 995-1008https://doi.org/10.1089/jpm.2015.0367 • Clark D • Baur N • Clelland D • et al. Mapping levels of palliative care development in 198 countries: the situation in 2017. J Pain Symptom Manage. 2020; 59 (e4): 794-807https://doi.org/10.1016/j.jpainsymman.2019.11.009 • Cardona-Morrell M • Kim JCH • Brabrand M • Gallego-Luxan B • Hillman K. What is inappropriate hospital use for elderly people near the end of life? A systematic review. Eur J Intern Med. 2017; 42: 39-50https://doi.org/10.1016/j.ejim.2017.04.014 • Donabedian A. Evaluating the quality of medical care. Milbank Memorial Fund Quarterly. 1966; 44: 166-206 • Stjernswärd J • Foley KM • Ferris FD. Integrating palliative care into national policies. J Pain Symptom Manage. 2007; 33: 514-520https://doi.org/10.1016/j.jpainsymman.2007.02.031 • The Economists Intelligence Unit The 2015 quality of death index: ranking palliative care across the world. Economist Intell Unit. 2021; (Accessed from:) • Steinhauser KE • Clipp EC • McNeilly M • Christakis NA • McIntyre LM • Tulsky JA. In search of a good death: observations of patients, families, and providers. Ann Intern Med. 2000; 132: 825-832 • Finkelstein EA • Bilger M • Flynn TN • Malhotra C. Preferences for end-of-life care among community-dwelling older adults and patients with advanced cancer: a discrete choice experiment. Health Policy. 2015; 119: 1482-1489 • Olmsted MG • Geisen G • Murphy J • Bell D • Morley M • Stanley M Methodology: U.S. News & World Report Best Hospitals 2015-16. US News & World Report's “Best Hospitals, 2015 (Accessed from:) • Bhadelia A • Oldfield LE • Cruz JL • Singh R • Finkelstein EA. Identifying the core domains to assess the ‘quality of death’: a scoping review. J Pain Symptom Manage. 2021; (submitted for publication) • Gonzalez-Sepulveda JM • Baid D • Johnson FR • Finkelstein EA. What is a good death? A choice experiment on care indicators for patients at end of life. J Pain Symptom Manage. 2021; (submitted for publication) 1. Bank W. World Data. The World Bank. Accessed from: https://data.worldbank.org/region/world • Programme UND The 2020 human development report. Human Dev Rep. 2020; (Accessed from:): 0969-4501 2. Bank W. GDP per capita, PPP (current international$). The World Bank. Accessed from: https://data.worldbank.org/indicator/NY.GDP.PCAP.PP.CD

3. World Health Organisation. Assessing the development of palliative care worldwide: a set of actionable indicators. World Health Organization. Accessed from: https://www.who.int/publications/i/item/9789240033351

• Connor SR
• Centeno C
• Garralda E
• Clelland D
• Clark D.
Estimating the number of patients receiving specialized palliative care globally in 2017.
J Pain Symptom Manage. 2021; 61: 812-816https://doi.org/10.1016/j.jpainsymman.2020.09.036