Evaluating Correlation and Interrater Reliability for Four Performance Scales in the Palliative Care Setting

Abstract 

Performance scales are used by clinicians to objectively represent a patient's level of function and have been shown to be important predictors of response to therapy and survival. Four different scales are commonly used in the palliative care setting, two of which were specifically developed to more accurately represent this population. It remains unclear which scale is best suited for this setting. The objectives of this study were to determine the correlations among the four scales and concurrently compare interrater reliability for each. Patients were each assessed at the same point in time by three different health care professionals, and all four scales were used to rate each patient. Spearman correlation coefficient values and both weighted and unweighted kappa values were calculated to determine correlation and interrater reliability. The results confirmed highly significant linear correlation among and between all four scales. Whether using a reliability measure that incorporates the concept of “partial credit” for “near misses” or a measure reflecting exact rater agreement, no one scale emerged as having a significantly higher likelihood of agreement among raters. We propose that what may be more important than clinical experience or rater profession is the level of training an individual health care professional rater receives on the administration of any particular performance scale. In addition, given that low levels of exact rater agreement could have substantial clinical implications for patients, we suggest that this parameter be considered in the design of future comparative studies.

Key Words: Performance scales, interrater reliability, correlation, palliative care