If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Address correspondence to: Céline Gélinas, RN, PhD, School of Nursing, McGill University, 3506 University Street, Wilson Hall, Montreal, Quebec H3A 2A7, Canada.
A repeated measure design was used to evaluate additional psychometric qualities (sensitivity and specificity) of the Critical-Care Pain Observation Tool (CPOT), a previously validated tool, in intubated intensive care unit (ICU) adults after cardiac surgery recruited in a university cardiology health center in Canada. Patients were evaluated while conscious and intubated (n=99/105), and extubated (n=105). For each of these two testing periods, patients were evaluated using the CPOT at rest (pre-exposure), during a nociceptive procedure-turning (exposure), and 20 minutes after the procedure (postexposure). The patients' self-reports of pain were obtained while intubated and extubated. During the nociceptive exposure, the CPOT had a sensitivity of 86%, a specificity of 78%, a positive likelihood ratio (LR+) of 3.87 (1.63–9.23), and a negative LR (LR−) of 0.18 (0.09–0.33) and was effective for the screening of pain. It also showed good specificity (83% and 97%) but lower sensitivity (47% and 63%) during nonexposure conditions. The CPOT cutoff score was >2 during the nociceptive exposure. After extubation, patients' self-reports of pain intensity were associated with the positive CPOT cutoff score previously determined. The CPOT adequately classified most of the patients with severe pain. The CPOT seems to be a useful tool to detect pain in intubated postoperative ICU adults, especially during a nociceptive procedure. Sensitivity and specificity of the CPOT need to be further explored during other nociceptive procedures and with different critically ill populations.
Although critical care clinicians strive to obtain the patients' self-report of pain, many factors compromise the patients' ability to communicate verbally. These include the use of sedative agents, mechanical ventilation, and the patients' change in level of consciousness.
have proposed that behaviors associated with pain are valuable proxies for self-report and should be considered as alternative measures of pain in nonverbal patients. Pain assessment methods must be adapted to conform to the communication capabilities of the patient. In the absence of patient self-report, behaviors become important indices for the assessment of pain.
Critically ill adults may not be able to communicate. Few pain assessment tools have been developed and tested in these patients: 1) Post-Anesthesia Care Unit Behavioral Pain Rating Scale
The PACU BPRS includes four behavioral indicators (restlessness, tense muscles, frowning or grimacing, and patient sounds) scored from 0 (absence of the behavior) to 3 (severe intensity of the behavior), for a possible total score ranging from 0 to 12. This tool showed good results for reliability (α=0.73 to 0.92) and criterion validity when compared with 30 and 36 extubated PACU patients' self-reports of pain intensity (r=0.56 to 0.80, P<0.05).
However, its measurement scale (absence to severe) remains subjective and can vary from one nurse to another, which may limit its clinical utilization.
includes 20 items clustered in a behavioral dimension (movements, vocalization, facial indicators, and posturing/guarding) and a physiological dimension (heart rate, blood pressure, respiratory rate, perspiration, and pallor). Each item is scored yes or no in this checklist. The PAIN was tested with 31 critically ill postoperative adult patients, most of them extubated. It showed acceptable criterion validity when the numbers of behavioral and physiological indicators observed were compared with the nurses' ratings of pain intensity (r=0.17 to 0.77, P<0.05). In a recent large study (Thunder Project II) conducted with 5957 critically ill adults, it was found that more behaviors (e.g., grimacing, rigidity, and vocalization) were exhibited by patients with procedural pain than without procedural pain.
includes three indicators (facial expression, movements of upper limbs, and compliance with the ventilator) scored from 1 to 4, for a possible total score ranging from 3 to 12. It was tested with three intensive care unit (ICU) mechanically ventilated and sedated adult samples: 30 trauma or postoperative patients,
with a weighted κ-coefficient of 0.74 and an intraclass correlation coefficient of 0.95, respectively. However, interrater agreement varied from low to high (36%–91%) in the study of Young and colleagues,
where lower agreement was obtained during pain assessments of nociceptive procedures. Discriminant validity was supported in all three studies where higher BPS scores were obtained with nociceptive procedures compared with rest or non-nociceptive procedures. The inclusion of “compliance with the ventilator” was an interesting indicator to consider for pain assessment in intubated patients. Finally, the operational definition for movements of upper limbs could be confused with muscle tension.
In summary, important efforts have been invested in research on the development of pain assessment tools in critically ill adults. Some of these tools showed limitations in their measurement scales or operational definition (PACU BPRS and BPS). The reliability and validity results for these tools were satisfactory. However, patient samples were not always representative of critically ill nonverbal adults (e.g., mechanically ventilated patients) and most sample sizes were small, limiting the generalizability of the findings. Also, criterion validity with the patients' self-reports of pain, which represents the most valid criterion, was not studied for some tools (PAIN and BPS). Except for the PAIN, for which no total intensity score can be calculated, sensitivity and specificity of the other existing tools were not documented.
To overcome those limitations, a new tool, the Critical-Care Pain Observation Tool (CPOT), was initially developed in French and forward-backward translated into English. It includes four behavioral categories: 1) facial expression, 2) body movements, 3) muscle tension, and 4) compliance with the ventilator for intubated patients or vocalization for extubated patients.
Items in each category are scored from 0 to 2 with a possible total score ranging from 0 to 8. Selection and operational definitions of the CPOT items were derived from the existing pain assessment instruments,
(trauma, postoperative, and medical cases), respectively. It demonstrated moderate to high interrater reliability (French version: weighted κ 0.52–0.88; English version: intraclass correlation coefficients 0.80–0.93). Discriminant validity was supported with higher CPOT scores during a nociceptive procedure (turning) compared with rest or a non-nociceptive procedure. For criterion validity, the patients' self-reports of pain intensity were associated with the CPOT scores (e.g., correlation coefficients from 0.40 to 0.71, P≤0.05). Based on those results,
the CPOT seems to be a useful tool for critically ill nonverbal patients. Further steps in tool validation are still necessary, however, including measurement of sensitivity and specificity, which provide relevant and valuable information to clinicians.
The analysis reported herein is derived from the larger study of Gélinas and colleagues
and explores the psychometric qualities of the CPOT by evaluating its sensitivity and the specificity in intubated ICU adults after cardiac surgery. As a complementary objective, the association between the CPOT score and the patients' self-reports of pain intensity after extubation is also described.
Methods
Design, Sample, and Ethics
A repeated measures design was chosen for this study, as described in a previous report.
A convenience sample of 105 cardiac surgery ICU patients from a cardiology health center in Canada was recruited. Patients were considered for inclusion if they were 18 years old or older, admitted for cardiac surgery, understood French, were in the ICU after surgery, and were able to hear and see. Patients were excluded if any of the following conditions were present: admitted for a heart transplant or thoracic aortic aneurysm repair, received medical treatment for chronic pain, had an ejection fraction less than 25%, had pre-existing psychiatric or neurological problems, had a dependence on alcohol or drugs, received neuromuscular blockade postoperatively, and had complications after the surgery (e.g., hemorrhage, delirium, and death).
The study was accepted by the human research committee of the health center. Recruitment was carried out the day before the surgery, and the study was explained to eligible patients. At this time, patients were taught how to use the pain intensity scales (Descriptive Pain Scale and Faces Pain Thermometer [FPT], see Measures section). The nurse research assistant explained to the patient the significance of both scales and the importance of rating their pain based on their own experience. Patients evaluated this teaching as essential to their care and supported that it should be provided to them before the surgery.
Measures
To allow determination of sensitivity and specificity of the CPOT, a gold standard criterion was necessary: the patients' self-reports of pain. We selected three verbal indicators: one for the patients while intubated and two for the patients after extubation. First, the yes or no response was obtained from intubated patients by head nodding for the presence or absence of pain. This pain criterion was selected because many intubated patients were unable to use pain intensity scales.
Patients after extubation used two different pain intensity scales: Descriptive Pain Scale and FPT. Two pain scales were chosen because the FPT was a newly developed tool
was essential to explore the validity of the FPT. Using a list of words of the Descriptive Pain Scale (none, mild, moderate, severe, unbearable), the patient was asked to choose the adjective that best described his/her current pain intensity. Finally, the FPT, developed for critically ill adults,
and other existing tools. The FPT was validated with the same 105 cardiac surgery ICU patients. The scale demonstrated good convergent validity (r=0.80 to 0.86, P<0.001 with the Descriptive Pain Scale described above) and discriminant validity (t=−5.10, P<0.001 comparing patient pain intensity at rest and during turning and yielding an association with a higher pain intensity score during turning). Content validity was also examined and patients positively evaluated its content and use.
Procedure
Six pain assessments with the CPOT, clustered into two testing periods, were completed in the patients' early postoperative period.
Each testing period included three assessments: at rest before the procedure (pre-exposure), during a nociceptive procedure-turning (exposure), and 20 minutes after the procedure (postexposure). Patient turning represented a previously confirmed nociceptive procedure known to be painful in critically ill adults.
The first testing period was completed while the patient was intubated and conscious. Consciousness was determined by a score of 2, 3, or 4 on the Ramsay Sedation Scale.
Finally, the second testing period was completed after extubation, approximately five hours after the first testing period.
For the two testing periods, patients were evaluated with the CPOT for one minute at rest (nonexposure), both before and after turning, and for the duration of the turning procedure (nociceptive exposure). This standardization of procedures was inspired by the work of Puntillo and colleagues.
The principal investigator and one critical care nurse who was trained to use the CPOT evaluated the patients. Interrater reliability was supported with moderate to high weighted κ-coefficients (0.52–0.88) for the six pain assessments.
On completion of the CPOT during the first testing period, intubated patients communicated the presence or absence of pain by head nodding (yes or no) to the question “Do you have pain?” During the second testing period, after completion of the CPOT, the extubated patients used the Descriptive Pain Scale and the FPT to grade their current pain intensity. To minimize bias, the CPOT evaluations were completed before asking the patient about their pain so that the CPOT raters were not influenced by the patient's self-report of pain.
Data Analysis
Data collected in the first testing period were used in a Receiver-Operating Characteristic (ROC) curve analysis to evaluate the ability of the CPOT to detect pain of intubated patients and to derive the threshold that maximizes both the sensitivity and specificity simultaneously.
Sensitivity refers to the true-positive rate. That is, a test or a tool is sensitive in detecting a disorder (i.e., pain) when it is actually present (i.e., as reported by patients). Specificity refers to the true-negative rate. In other words, the test should be negative in patients who reported having no pain. The ROC curve is a plot of sensitivity (the true-positive fraction) against 1 minus specificity (the false-positive fraction) pairs resulting from different values of the CPOT (0–8). Perfect discrimination has a plot that passes through the upper left corner of the plot where the true-positive fraction is 100% and the false-positive fraction is 0. The closer the plot is to the upper left corner of the curve, the higher the overall predictive power of the criterion. The area under the curve (AUC) provides a measure (on a scale from 0.5 to 1 where 0.5 represents discrimination no better than chance and 1 perfect discrimination) of the CPOT's ability to distinguish between patients who reported having pain (yes) and those who reported not having pain (no). It was tested if the area under the ROC curve was greater than 50% using an alpha level of significance of 0.05. Using the threshold suggested by the ROC curve, sensitivity and specificity of the CPOT were calculated.
Likelihood ratios (LRs), which are alternative statistics for summarizing diagnostic accuracy, were also calculated to evaluate the ability of the threshold CPOT value to “rule in” (LR positive [LR+]) or “rule out” (LR negative [LR−]) the presence or the absence of pain reported by intubated patients. An LR+ describes how much more likely a positive test result is to be found in intubated patients who reported having pain compared with those who reported having no pain. A value greater than 1 is expected for LR+. An LR− describes how many times more likely a negative test result is to be found in intubated patients who reported having no pain compared with those who reported having pain. A value less than 1 is expected for LR−.
Finally, the association between the CPOT's threshold value and the extubated patients' self-reports of pain intensity obtained at the second testing period was examined through use of Chi-square tests. This association was tested using the Descriptive Pain Scale and the FPT, separately. To perform the test, patients' self-reports of pain intensity on the FPT were clustered into four categories:
a total of 131 patients were approached for consent the day before surgery, and 117 (89%) agreed to participate in the study. Reasons for refusal were 1) anxious about the surgery (n=9), 2) not interested (n=3), 3) undecided (n=1), and 4) bad experience with research (n=1). During the course of the study, eight patients were excluded due to postoperative complications (hemorrhage, delirium, and death); three due to surgery cancellation; and one due to extubation right after surgery. The final sample size was 105 patients enrolled over a three-month time period. Eighty-three were males (79%) and 22 were females (21%), with a mean age of 60 years (standard deviation=8). Most patients underwent coronary artery bypass graft (CABG, 79%), some valvular surgery (10.5%), both CABG and valvular surgery (8.6%), or interauricular/interventricular communication repair (1.9%). All patients had sternal incisions. Patients were receiving continuous infusions of fentanyl when they were admitted to the ICU from surgery. The fentanyl decreased in amount from 65.1±29.0 μg/hour in intubated conscious patients to 50.7±31.1 μg/hour after extubation. Intravenous bolus of fentanyl was not part of the pain protocol and was only available on an individual basis (physician's prescription). Only four patients received a bolus of fentanyl before turning.
Data collection was completed on 99 of the 105 awake intubated patients. The remaining six patients were extubated before the completion of this testing period. Finally, all 105 patients were assessed once extubated.
Sensitivity and Specificity of the CPOT
More than 50% of intubated patients reported pain while at rest. Being in pain was observed at pre- and postexposure. This percentage was higher (80%) during nociceptive exposure (see Table 1). ROC curve analysis was performed at these three assessments (see Fig. 1, Fig. 2, Fig. 3). During and after the nociceptive exposure, AUC indicated intermediate to good discriminative properties (i.e., AUC of 0.75–0.85). During nociceptive exposure, the threshold associated with maximization of the sums of sensitivity and specificity was found to be a score greater than 2 on the CPOT, whereas for the nonexposure periods (pre- and postexposure), the threshold was found to be a score greater than 1 on the CPOT. Properties of the CPOT for the detection of pain in intubated patients are presented in Table 2. At these thresholds, sensitivity was higher than specificity during the nociceptive exposure. In contrast, specificity was higher than sensitivity during nonexposure (pre- and postexposure). At pre-exposure and during nociceptive exposure, a patient who reported to be in pain had a three to four times greater chance (LR+) for a positive score on the CPOT (>2) than a patient who reported no pain. The LR+ was much higher at postexposure. Finally, the accuracy rate of the CPOT to detect presence or absence of pain in intubated patients was good during nociceptive exposure and at postexposure, but was lower at pre-exposure. In summary, the CPOT was effective in showing good detection properties, particularly during and after the nociceptive exposure, when pain could be higher.
Table 1Intubated Patients' Self-Report of Pain (Yes or No) at the First Testing Period
Assessment
Yes, Presence of Pain n (%)
No, Absence of Pain n (%)
Pre-exposure
53 (56.4)
41 (43.6)
Nociceptive exposure
79 (81.4)
18 (18.6)
Postexposure
54 (58.1)
39 (41.9)
Because of intermittent drowsiness postoperatively, five patients at pre-exposure, two during nociceptive exposure, and six at postexposure were unable to give their self-reports of pain by head nodding.
Association Between the Extubated Patients' Self-Reports of Pain Intensity and the CPOT Threshold Value
Extubated patients' self-reports of pain intensity were associated with the CPOT threshold value (a score greater than 1 on the CPOT at rest (pre- and postexposure) and a score greater than 2 on the CPOT during nociceptive exposure) previously determined while the patients were intubated (see Table 3). Chi-squared tests were significant for the three assessments (pre-exposure, nociceptive exposure, and postexposure) for both pain intensity scales. It was observed that the CPOT was able to adequately classify most patients with severe or unbearable pain. At pre-exposure, seven patients with severe or unbearable pain out of 10 (70%) on the Descriptive Pain Scale and seven out of 11 (64%) with severe pain on the FPT obtained a score >1 on the CPOT. During the nociceptive exposure, 21 patients with severe or unbearable pain out of 25 (84%) on the Descriptive Pain Scale and 17 out of 20 (85%) with severe pain on the FPT obtained a score >2 on the CPOT. Finally, at postexposure, results were weaker. Four patients with severe pain out of six (67%) on the Descriptive Pain Scale and only one patient out of five (20%) with severe pain on the FPT obtained a score >1 on the CPOT.
Table 3Association Between the Extubated Patients' Self-Report of Pain Intensity and CPOT Threshold Value at the Second Testing Period
This report described additional psychometric qualities (i.e., sensitivity and specificity) from a larger validation study of the CPOT, a behavioral pain assessment tool developed for clinical use in critically ill intubated adults.
The specificity of the CPOT was high for all assessments of intubated and conscious patients: the higher the specificity of a test or a tool, the lower the false-positive cases.
Thus, the risks of administering an analgesic to a patient who does not need it and the potential adverse effects that could result from the medication would be avoided.
Sensitivity was high during the nociceptive exposure, moderate at postexposure, and low at pre-exposure. A possible explanation would be that, when patients are not exposed to a nociceptive procedure, some of them do not show or are able to control their behaviors even if they feel pain. This might be different when they are exposed to a nociceptive procedure. That is, they might be unable to control reflexive pain behaviors, which would explain the higher sensitivity during turning (nociceptive exposure). Patients with a positive CPOT (>2) score indicating the presence of pain during a nociceptive procedure would benefit from receiving an analgesic. Indeed, administering an analgesic before a nociceptive procedure, such as turning, may not be harmful and could prevent procedural pain.
All patients in this study received continuous infusions of fentanyl, but rarely (n=4) received an intravenous bolus before turning; this precluded exploration of the impact of this intervention on the CPOT findings.
The positive CPOT threshold value differed according to the exposure status. It was lower when not exposed (>1) and higher during exposure (>2). These results may be explained in two ways. First, intubated patients may react more intensely to pain when they are exposed to a nociceptive procedure (turning) than while they are at rest. This is not surprising because turning may cause additional pain to these patients. In a previous study, turning was identified as the most painful of six procedures performed on acutely and critically ill adults.
Second, the nurse may perceive the presence of more behaviors during patient turning if she knows the procedure is painful. So, if the nurse thinks her patient can tolerate turning and observes a high score on the CPOT, she may give an analgesic bolus before the next turning procedure and also right after the procedure.
For extubated patients, an association was found between patients' self-reports of pain intensity and the CPOT. The more severe the patients' pain, the more likely a positive result on the CPOT was found. In a meta-analysis, Labus and colleagues
found that pain behaviors were positively and moderately associated with patients' self-reports of pain intensity. Also, in two previous studies of Gélinas and colleagues,
the patients' self-reports of pain intensity were moderately to highly correlated with the CPOT scores (r=0.40–0.71, P<0.05). Even if pain behaviors are known to be related to the patients' self-reports of pain intensity, it must be emphasized that these two indicators of pain are different in nature (behavioral and sensory components of pain) and may be complementary. Thus, it is not surprising to find that some patients with moderate or mild pain obtained a positive score (>1 at pre- and postexposure and >2 during nociceptive exposure) on the CPOT (false-positive cases), and that other patients with severe pain obtained a negative score on the CPOT (false-negative cases). As highlighted in the clinical recommendations of the American Society for Pain Management Nursing,
the use of behaviors seems to be more relevant for the detection of the presence of pain than for the interpretation of pain levels (mild, moderate, and severe). In fact, the CPOT was more effective in adequately classifying patients with severe pain than those with mild or moderate pain. Finally, it must be highlighted that patients were intubated during the first testing period and were extubated during the second testing period. Thus, the patients' clinical conditions had changed as they recovered postoperatively. That CPOT thresholds may differ between intubated and extubated patients is an interesting possibility that warrants further exploration.
The potential limitations of this study must be considered. The postoperative pain experience may be different for patients with other conditions, which could alter study results. Also, the gold standard (presence or absence of pain: yes or no) used for evaluation of sensitivity and specificity of the CPOT in intubated patients was limited because it did not allow us to describe the degree of the patients' pain intensity. Moreover, postoperative drowsiness led to missing data for some patients. Finally, potential bias of the investigator and her colleague are possible. More raters should be used in subsequent evaluations of the CPOT.
Although the patient's self-report represents the most valid measure for pain, it may be impossible to use in many critically ill patients. As already noted by Anand and Craig,
for critically ill nonverbal adults, pain behaviors are valuable forms of self-report and should be considered as alternative measures of pain. The change in these behaviors after analgesic therapy should also be assessed. Thus, the use of the CPOT may expedite pain management in patients with difficulties in communication.
In future work, the sensitivity and the specificity of the CPOT need to be studied in different critical care populations and associated with a variety of nociceptive procedures. Inclusion of other behavioral or physiological indicators in the tool could also be considered for testing.
Conclusions
The specificity of the CPOT was high across all conditions (no exposure and nociceptive exposure), but sensitivity was high only during nociceptive exposure. Sensitivity and specificity of behavioral indicators need to be further explored in different critically ill populations and various situations or procedures. Development of reliable and valid pain assessment tools to be used for patients who are not able to self-report is an essential first step toward better pain management for this vulnerable group of critically ill patients. The CPOT is a valid tool and various psychometric qualities have been examined. Available in English and French, it is already being used for research or clinical purposes in North America and requests to translate the tool in other languages were recently made.
This work was supported by a Postdoctoral Fellowship from the Fonds de recherche en santé du Québec and the Fondation de recherche en sciences infirmières du Québec.
00290-X&id=gr1.gif){kind=link}
00290-X&id=gr2.gif){kind=link}
00290-X&id=gr3.gif){kind=link}