Measurement properties of ID-PALL, a new instrument for the identiﬁcation of patients with general and specialized palliative care needs

Context : To improve access to palliative care, identification of patients in need of general or specialized palliative care is necessary. To our knowledge, no available identification instrument makes this distinction. ID-PALL is a screening instrument developed to differentiate between these patient groups. Objective : To assess the structural and criterion validity and the inter-rater agreement of IDPALL. Methods : In this multicenter, prospective, cross-sectional study, nurses and physicians assessed medical patients hospitalized for 2 to 5 days in two tertiary hospitals in Switzerland using ID-PALL. For the criterion validity, these assessments were compared to a clinical gold standard evaluation performed by palliative care specialists. Structural validity, internal consistency and inter-rater agreement were assessed. Results : 2232 patients were assessed between January and December 2018, 97% by nurses and 50% by physicians. The variances for ID-PALL G and S are explained by two factors, the first one explaining most of the variance in both cases. For ID-PALL G, sensitivity ranged between 0.80 and 0.87 and specificity between 0.56 and 0.59. ID-PALL S sensitivity ranged between 0.82 and 0.94, and specificity between 0.35 and 0.64. A cut-off value of 1 delivered the optimal values for patient identification. Cronbach’s alpha was 0.78 for ID-PALL G and 0.67 for ID-PALL S. The agreement rate between nurses and physicians was 71.5% for ID-PALL G and 64.6% for ID-PALL S. Conclusion : ID-PALL is a promising screening instrument allowing the early identification of patients in need of general or specialized palliative care. It can be used by nurses and physicians without a specialized palliative care training. Further testing of the finalized clinical version appears warranted. This multicenter study demonstrates structural and criterion validity of ID-PALL, a new brief instrument for the interprofessional identification of patients in need of both general and specialized palliative care.


Introduction
Depending on care settings, illnesses, co-morbidities and frailty, reported prevalence of palliative care patients varies between 7% and 73%. Despite the development of palliative care worldwide, access remains limited, especially for non-cancer patients, older people, and vulnerable and cultural minority patients (1)(2)(3)(4). It is an ethical obligation to provide palliative care access to all patients irrespective of their diagnoses (5). International recommendations emphasize the necessity for systematic identification of patients' palliative care needs in order to ensure equitable access to care and promote social justice (5,6). Although the World Health Organization (WHO) definition clarifies the concept of palliative care (7), it does not make the distinction between general and specialized palliative care needs (8). General palliative care refers to patients with a life-threatening prognosis of a chronic, progressive, incurable and life-threatening disease, or who have reached the end of their life, without complex problems. About 75% to 80% of all palliative care patients do not have complex needs and thus do not require specialists in palliative care (8)(9)(10). The remaining 20% to 25% of palliative care patients have complex needs, including unstable clinical conditions, unpredictability and/or high level of bio-psycho-socio-existential suffering, requiring specific treatment (11,12). Although the distinction is being made between generalized and specialized palliative care, there is still significant debate about where these thresholds lie.
For the moment, in clinical practice, this distinction is relatively arbitrary and subject to misinterpretation (8,11,13). However, appropriate differentiation between these patient groups is pre-requisite for adequate delivery of palliative care (14,15).
Instruments for the identification of palliative care patients are mostly screening checklists (16)(17)(18). The purpose of a screening test is to estimate the likelihood of having the condition being screened (19,20). Diagnostic tests, on the other hand, make it possible to detect or exclude a pathology in a patient (21). Screening instruments should encourage discussion among professionals about what decision to make, be inexpensive and easy to use (21). For a screening instrument, it is recommended to maximize sensitivity to avoid missing patients with the condition (true positives) (21)(22)(23).
A number of instruments to identify patients in need of palliative care are available. In a recent systematic review (24,25) we found four instruments designed for hospital use, namely the CAPC (26), the NECPAL (27), the PCST (28) and the SPICT (29). None exhibited sufficiently valid psychometric measurement properties according to the COSMIN criteria (30). Criterion validity, when assessed, was based on mortality prediction (31)(32)(33)(34), which is incongruent with the concept of early integration of palliative care (16). Finally, none of these instruments differentiates between patients in need of general versus specialized palliative care (16,35).
To fill in this gap, we developed ID-PALL © (IDentification of patients in need of PALLiative care) a new screening instrument developed as a clinician-reported outcome measure (for details of the instrument development, see ref. 36). ID-PALL does not require detailed medical information and can be used by nurses and physicians alike based on their clinical observations. The content and face validity were deemed to be acceptable by both the expert and the target population (36). The aim of this study was to assess the structural validity, the criterion validity and the inter rater agreement of ID-PALL.

Study setting
This multicenter, prospective and cross-sectional study was conducted in the French and Italian linguistic regions of Switzerland. To obtain a homogenous sample, it was carried out in seven internal medicine units: four in a university French-speaking hospital and three in a community Italian-speaking hospital, for a total of 190 beds. Both hospitals have a dedicated inpatient palliative care unit and a palliative care hospital support team.

Instrument
ID-PALL includes two parts: part one (seven items) to identify patients requiring general palliative care (ID-PALL G) and part two (eight items) to identify patients requiring specialized palliative care (ID-PALL S). Hereinafter, the name ID-PALL refers to the total item list used for the validation process. Items assess important aspects of palliative care, such as presence of a progressive illness, cessation of life-sustaining treatment, and psychosocial or existential distress of patients or relatives. The level of complexity distinguishes the two parts. ID-PALL S is only completed if ID-PALL G is positive. Each item is rated based on a structured "yes/ no" response format. In each part, one positive response to any item was considered as the cut-off to require either general or specialized palliative care. Nurses and physicians were instructed to fill in the instrument based on their clinical observations, anamnesis and when possible the patient's self-assessment.
Specific written recommendations were developed for patients requiring general palliative care, for the benefit of non-specialized healthcare professionals caring for the patients. Their acceptance and impact will be assessed in a separate study.

Ethical considerations
Due to the sensitive nature of the study topic, the local Human Research Ethics Committee stipulated that ID-PALL had to be used by mixing all the items, rather than in its definitive two-part format, so as not to influence clinical practice. Thus, healthcare professionals completing ID-PALL were unaware of its original structure. Two regional human research ethics committees approved this study on November 6, 2017.

Structural validity
Structural validity was tested separately for ID-PALL G and ID-PALL-S. The Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy (which may vary between 0 [no adequacy] and 1 [maximal adequacy]; a value of 0.6 is generally accepted as sufficient) and Bartlett's test of sphericity (a measure of the correlation matrix between the items) were used to assess the suitability of the data for an exploratory factorial analysis (EFA). The internal factor structure of ID-PALL G and ID-PALL S was tested with a principal component analysis using varimax rotation. Eigenvalues higher than one were used to verify factor solution accuracy (Kaiser criterion). Confirmatory factor analysis (CFA) is recommended in order to obtain a 'very good' COSMIN rating. However, an EFA is rated as "adequate" (30). In order to keep only the factors with the greatest variance, we have retained those with an eigenvalue ≥1.

Internal consistency
According to COSMIN recommendations, we calculated Cronbach's alpha. Sufficient internal consistency is assumed for Cronbach's alpha ≥ 0.70 (30).

Criterion validity
Criterion validity evaluates the extent to which the assessed measurement is associated with a recognized external criterion, or gold standard (37,38). However, "this is a challenging concept for health care services that are not disease-based, such as palliative care" (39). In the absence of a recognized gold standard, the use of expert clinical assessment is the best alternative (40). Consequently, and in line with the interprofessional approach of the discipline, we have considered the combined clinical judgment of a physician-nurse dyad, both specialized in palliative care, as the "clinical gold standard". Its role was to classify the patients into one of the following categories: not requiring palliative care, requiring general palliative care, or requiring specialized palliative care.

Evaluation based on ID-PALL
Primary care nurses assessed all patients meeting the study criteria at both sites. Twice a week, the research team gave the nurses an ID-PALL form with the name of the patients to be evaluated.

Statistical analyses and sample size calculation
We used Area Under the receiver operator Curve (AUC) analysis to show the screening ability of ID-PALL G in identifying general palliative care patients against non-palliative care patients, and of ID-PALL S in identifying specialized palliative care patients against general palliative care patients. For evidence of good measurement properties, the AUC should be ≥ 0.70 (30). We also calculated the sensitivity, specificity, and the positive and negative predictive values (PPV and NPV) for different cut-off scores.
The sample size was calculated according to the prevalence of patients in need of general or specialized palliative care reported in the literature (around 35% for general palliative care and 7% for specialized palliative care in internal medicine units) (41,42).The sensitivity threshold was set at 0.9 and the precision level at 5%. A sample size of 2000 patients was calculated in order to be able to identify at least 700 general palliative care patients.
Considering our large sample, we chose to remove all assessments with missing data in order to avoid any misinterpretation. Data from the two sites did not show major differences and were pooled together. For the results by site, see supplementary material.

Inter-rater agreement
Because we assessed the agreement between nurses and physicians, the term inter-rater agreement appears to be more appropriate than inter-rater reliability. The two professions have complementary competences and different training backgrounds, thus differences in assessment are to be expected. Cohen's kappa (0 = same concordance as that due to chance; 1 = perfect concordance) and percentage of agreement were calculated.

Procedure
To assess the inter-rater agreement, 85 general palliative patients needed to be assessed by nurses and physicians. This sample size was applied for both study sites. Thus, primary care physicians at the university hospital were involved in the last three months of data collection.
At the community hospital, primary care physicians were involved in the assessment for the entire data collection time. Primary care nurses and physicians completed ID-PALL of the same patient on the same day on an individual ID-PALL form, independently, at both sites.
Statistical analyses were carried out using SPSS 26.

ID-PALL availability in different languages
The instrument was tested in its original language (French) and in the translated Italian version. A standardized forward-backward translation process was used to translate the instrument in Italian, German and English (43)

Recruitment and sample characteristics
Between January and December 2018, 2232 of the 2479 patients (90%) who met the inclusion criteria were assessed with ID-PALL (1533 at the university hospital and 699 at the community hospital). Nurses assessed 97% of the included patients and physicians 50% ( Figure 1). Mean age of the patients was 73 years (±16.5); 54% were men. Patients from the university hospital were significantly older than patients from the community hospital (75.3 vs 71.2 yrs.). Patients had a large spectrum of pathologies, with a strong representation of chronic heart failure and cancer, which are distributed differently between the two hospitals (Table 1).

Structural validity
For ID-PALL G, the KMO measure indicated a sampling adequacy of 0.834 and a Bartlett's test of sphericity for the correlation between items at  2 = 3265.276 (p = .000). For ID-PALL G, the exploratory factor analysis with varimax rotation revealed two factors explaining 59% of the total variance for ID-PALL G with an eigenvalue higher than one (3.106 and 1.029).
The first factor explained the greatest percentage of this variance (44% vs 15%).
For ID-PALL S, the KMO measure indicated a sampling adequacy at .786 and a Bartlett's test at  2 = 2437.381 (p = .000). The factorial analysis revealed two factors explaining 47% of the total variance for ID-PALL S with an eigenvalue higher than one (2.720 and 1.062). The first factor explained the greatest percentage of this variance (34% vs 13%). Correlations between items and each factor for ID-PALL G and S are shown in Table 2. Insert Table 2 here

Internal consistency
Cronbach's alpha for ID-PALL G was 0.78 and for ID-PALL S 0.67.    Tables 3 and 4 show the sensitivity, specificity, positive and negative predictive values for the various cut-off scores for ID-PALL G and ID-PALL S by considering the nurses' assessments. In both cases, the cut-off value of 1 delivers the optimal values for patient identification. Table 3: ID-PALL G score cut-offs to identify general palliative care patients by nurses, N = 1920

Criterion validity
Insert Table 3 here Table 4: ID-PALL S score cut-offs to identify specialized palliative care patients by nurses, N = 806 Insert Table 4 here The score of 1 is also the best cut-off for the physicians when considering both the ID-PALL

Inter-rater agreement
Cohen's kappa was calculated on 877 patients for ID-PALL G and 478 for ID-PALL S. The results were 0.39 with an agreement rate of 70.5% for ID-PALL G and 0.21 with a 61.5% agreement for ID-PALL S.

Discussion
This multicenter study involving over 2200 patients is, to the best of our knowledge, the first to validate a screening instrument that distinguishes between general and specialized palliative care. Nurses on the wards were able to assess 97% of all patients meeting the study criteria, which demonstrates the feasibility of the assessment in clinical practice.

Structural validity of ID-PALL
While two factors explain 59% of the variance for ID-PALL G and 47% for ID-PALL S, the first factor explains the greater variance for both parts of the instrument. When observing the items presenting the highest correlations with the first factor, it is notable that they are characterized by a certain level of heterogeneity in terms of content, as detailed in Table 2.
This variety of dimensions gathered under an overarching factor is in line with the preestablished conceptualization of ID-PALL, which strived to encompass as much as possible the multidimensionality of palliative care. The results of the factor analysis led us not to apply specific dimensions in each of the two parts of the ID-PALL, but rather to consider all the items for an overall score. Thus, we used this overall score to assess the criterion validity of our instrument.
When comparing our results with those of other instruments like the NECPAL (27) and the SPICT (29) which were designed for hospital use, it is worth mentioning that the underlying structure of these two instruments includes general health status criteria as well as specific criteria related to the underlying diagnoses. The latter criteria are not required to complete ID-PALL, in order to make it easier for nursing professionals to use it independently of physicians.
According to our EFA results, we could consider removing items 5 (relatives' psychosocial suffering) and 6 (palliative sedation) of ID-PALL. However, it seems essential to keep item 5 to remain coherent with the definition of palliative care which includes patients and their relatives and which encourages psychosocial care for both. Item 6 may be considered redundant in light of items 1 (severe and persistent symptom) and 4 (severe psychosocial or existential suffering). As the EFA was conducted using the items in a mixed manner and not on the final version of the instrument, we chose to wait for the CFA which will be carried out on the final version of the instrument to determine whether or not to keep these items.

Internal consistency
According to the COSMIN criteria, the internal consistency of ID-PALL G is considered as a good measurement property. The internal consistency of ID-PALL S is slightly under this cutoff and remains the same even if an item is removed. Our instrument is based on a formative model where the different items are heterogeneous with respect to the construct. This is different than a reflective model in which all items are a manifestation of the same underlying construct and are highly correlated (37). Thus, we are satisfied with the internal consistencies found for both ID-PALL parts. Our systematic review revealed that none of the included instruments had a measure of internal consistency, which does not allow for a comparison.

Criterion validity of ID-PALL
According to the COSMIN criteria, the correlation with the gold standard demonstrated good measurement properties for ID-PALL G and S. (30). For both parts of the instrument, we found the cutoff of 1 to be the optimal score when searching for an adequate balance between sensitivity and specificity. However, while the NPV appears good for both parts of the ID-PALL, the PPV is low for the ID-PALL G and very low for the ID-PALL S (55.8% and 28.2%, respectively). Of note, these low PPV scores are coherent with other studies that have assessed the criterion validity of various screening instruments in palliative care (31,44,45).
In terms of sensitivity and specificity, ID-PALL values are quite similar to those reported for previously developed instruments concerning their ability to predict deaths at 12 months (NECPAL: sensitivity 0.91, specificity 0.32 (46); SPICT 0.84 and 0.58, respectively (47)).
However, the use of mortality at 12 months as gold standard to assess the validity of these instruments represents a major limitation, since the clinical need for palliative care is evidently neither dependent on nor necessarily correlated with 12-month mortality (16).
The choice of a cut-off at 1 aims at maximizing the sensitivity of the instrument (20), in order not to miss patients with palliative care needs and to provide them with the appropriate level of palliative care as early as possible in their illness trajectory. Moreover, preliminary observations from an ongoing feasibility study show that the systematic use of such an instrument for all admitted patients may represent an idealistic goal in acute hospital settings.
Clinicians seem more likely to use ID-PALL to confirm or disprove their clinical hypotheses for each individual patient. We therefore recommend the use of ID-PALL as an instrument to support clinical decision making rather than a "classical" screening method (21,48) According to the COSMIN recommendations and in the absence of a generally accepted gold standard, we have tested ID-PALL using a gold standard based on clinical assessment completed by a nurse and physician dyad specialized in palliative care (49). This choice reflects the fundamentally interprofessional nature of palliative care, and is in line with the aim of developing an instrument that can be used by nurses and physicians interchangeably.

Inter-rater agreement
The inter-rater agreement between nurses and physicians was low to moderate, and physicians tended to identify general palliative care patients slightly better than nurses. This may be due to medical training in Switzerland including mandatory teaching in palliative care, while this is optional in pre-graduate nursing training. In addition, a better knowledge of the medical diagnosis may still positively influence the ability to complete ID-PALL G. In order to base the assessment on a comprehensive knowledge of the patients and families. We encourage completing ID-PALL based on an interprofessional discussion.
In total, three of the eight categories of the COSMIN criteria were tested in this study. To be in line with the COSMIN methodology, further tests will have to be carried out (e.g. measurement error, cross-cultural validity, responsiveness). Consistent with the development phase of ID-PALL, we began with an EFA. A CFA will be conducted as part of a feasibility study, which is currently underway. Because of the limits of the inter-rater measurement mentioned above, we plan to re-assess inter-rater reliability within each profession. In the future, it would also be interesting to test ID-PALL in comparison with other identification instruments measuring similar constructs.

Strength and limitations
The main strength of this study is its large sample size of over 2200 patients. One major limitation is that the clinical gold standard of the university hospital relied on electronic patient records, which often lacked information about social, psychological and spiritual aspects. In addition, due to the decision of the ethics committee, we had to perform the study using a list of mixed-up items instead of ID-PALL G and S in their final format. We can thus hypothesize that results may improve when professionals will use the instrument in its intended format, as filling in the individual items of the instrument without the understanding of its clinical logic may have complicated this process for the study participants. Finally, we could not collect demographic data from the participating professionals because it was requested from the medical and nursing heads that the collaborators not be identified in order to protect their confidentiality, maximize study participation and compliance with the research protocol.

Conclusion
ID-PALL is, to the best of our knowledge, the first screening instrument allowing the early identification of patients in need of general vs. specialized palliative care by nurses and physicians without a specialized palliative care training. The structural validity of the two parts of ID-PALL is good and the choice of a cut-off at 1 for both parts of the instrument is confirmed. We tested the criterion validity of ID-PALL against a clinical gold standard evaluationa new approach for this kind of instruments. This multicenter study shows that ID-PALL is a promising instrument with sufficiently robust measurement properties to warrant further testing in a clinical implementation study, which is currently underway. P. Larkin, E. Poncin and C. Rime for the English forward-backward translation of ID-PALL.
Finally, our heartfelt thanks go to all nurses and physicians who have completed ID-PALL for this study.

Data management and sharing
All data of the study can be obtained from the first author.

Declaration of conflicts of interest
The authors have no competing interests to declare.

Funding
The Swiss Academy for Medical Sciences funded this study through its "Research in palliative care" program. The study was also supported by the Public health service of the Canton de Vaud and Ticino, as well as by the Palliative and supportive care service, the Training center and the Head of nursing of the Lausanne University Hospital.

Supplementary material
Supplementary material for this article is available online and includes the full English version of ID-PALL.