Advertisement

Sensitivity and Specificity of the Critical-Care Pain Observation Tool for the Detection of Pain in Intubated Adults After Cardiac Surgery

Open AccessPublished:July 03, 2008DOI:https://doi.org/10.1016/j.jpainsymman.2007.12.022

      Abstract

      A repeated measure design was used to evaluate additional psychometric qualities (sensitivity and specificity) of the Critical-Care Pain Observation Tool (CPOT), a previously validated tool, in intubated intensive care unit (ICU) adults after cardiac surgery recruited in a university cardiology health center in Canada. Patients were evaluated while conscious and intubated (n=99/105), and extubated (n=105). For each of these two testing periods, patients were evaluated using the CPOT at rest (pre-exposure), during a nociceptive procedure-turning (exposure), and 20 minutes after the procedure (postexposure). The patients' self-reports of pain were obtained while intubated and extubated. During the nociceptive exposure, the CPOT had a sensitivity of 86%, a specificity of 78%, a positive likelihood ratio (LR+) of 3.87 (1.63–9.23), and a negative LR (LR) of 0.18 (0.09–0.33) and was effective for the screening of pain. It also showed good specificity (83% and 97%) but lower sensitivity (47% and 63%) during nonexposure conditions. The CPOT cutoff score was >2 during the nociceptive exposure. After extubation, patients' self-reports of pain intensity were associated with the positive CPOT cutoff score previously determined. The CPOT adequately classified most of the patients with severe pain. The CPOT seems to be a useful tool to detect pain in intubated postoperative ICU adults, especially during a nociceptive procedure. Sensitivity and specificity of the CPOT need to be further explored during other nociceptive procedures and with different critically ill populations.

      Key Words

      Introduction

      Pain is recognized as a subjective and multidimensional experience.
      • Loeser J.D.
      • Cousins M.J.
      Contemporary pain management.
      Its subjective characteristic reinforces the fact that pain is whatever the experiencing person says it is and exists whenever he or she says it does.
      • McCaffery M.
      Nursing management of the patient with pain.
      Pain is also defined as a stressor for many patients in critical care settings,
      • Rotondi A.J.
      • Chelluri L.
      • Sirio C.
      • et al.
      Patients' recollections of stressful experiences while receiving prolonged mechanical ventilation in an intensive care unit.
      and it is not unusual for the intensity of this pain to be described as moderate to severe.
      • Gélinas C.
      Management of pain in cardiac surgery ICU patients: have we improved over time?.
      • Puntillo K.A.
      • White C.
      • Morris A.B.
      • et al.
      Patients' perceptions and responses to procedural pain: results from Thunder Project II.
      • Stanik-Hutt J.
      • Soeken K.L.
      • Fontaine D.K.
      • Gift A.G.
      Pain experiences of traumatically injured patients in a critical care setting.
      Although critical care clinicians strive to obtain the patients' self-report of pain, many factors compromise the patients' ability to communicate verbally. These include the use of sedative agents, mechanical ventilation, and the patients' change in level of consciousness.
      • Hamill-Ruth R.J.
      • Marohn L.
      Evaluation of pain in the critically ill patient.
      • Shannon K.
      • Bucknall T.
      Pain assessment in critical care: what have we learnt from research.
      Self-report is considered as the most valid measure of pain.
      • Agency for Health Care Policy and Research (AHCPR)
      Acute pain management: operative or medical procedures and trauma, part 1.
      But how should we assess pain in patients who cannot communicate their self-report? Anand and Craig
      • Anand K.J.S.
      • Craig K.D.
      New perspectives on the definition of pain.
      have proposed that behaviors associated with pain are valuable proxies for self-report and should be considered as alternative measures of pain in nonverbal patients. Pain assessment methods must be adapted to conform to the communication capabilities of the patient. In the absence of patient self-report, behaviors become important indices for the assessment of pain.
      • Herr K.
      • Coyne P.J.
      • Key T.
      • et al.
      Pain assessment in the nonverbal patient: position statement with clinical practice recommendations.
      Critically ill adults may not be able to communicate. Few pain assessment tools have been developed and tested in these patients: 1) Post-Anesthesia Care Unit Behavioral Pain Rating Scale
      • Mateo O.M.
      • Krenzischek D.A.
      A pilot study to assess the relationship between behavioral manifestations and self-report of pain in postanaesthesia care unit patients.
      (PACU BPRS), 2) Pain Assessment and Intervention Notation
      • Puntillo K.A.
      • Miaskowski C.
      • Kehrle K.
      • et al.
      Relationship between behavioral and physiological indicators of pain, critical care patients' self-reports of pain, and opioid administration.
      (PAIN), and 3) Behavioral Pain Scale
      • Payen J.F.
      • Bru O.
      • Bosson J.L.
      • et al.
      Assessing pain in critically ill sedated patients by using a behavioral pain scale.
      (BPS).
      The PACU BPRS includes four behavioral indicators (restlessness, tense muscles, frowning or grimacing, and patient sounds) scored from 0 (absence of the behavior) to 3 (severe intensity of the behavior), for a possible total score ranging from 0 to 12. This tool showed good results for reliability (α=0.73 to 0.92) and criterion validity when compared with 30 and 36 extubated PACU patients' self-reports of pain intensity (r=0.56 to 0.80, P<0.05).
      • Mateo O.M.
      • Krenzischek D.A.
      A pilot study to assess the relationship between behavioral manifestations and self-report of pain in postanaesthesia care unit patients.
      • Webb M.R.
      • Kennedy M.G.
      Behavioral responses and self-reported pain in postoperative patients.
      However, its measurement scale (absence to severe) remains subjective and can vary from one nurse to another, which may limit its clinical utilization.
      • Feinstein A.R.
      Clinimetrics.
      The PAIN
      • Puntillo K.A.
      • Miaskowski C.
      • Kehrle K.
      • et al.
      Relationship between behavioral and physiological indicators of pain, critical care patients' self-reports of pain, and opioid administration.
      includes 20 items clustered in a behavioral dimension (movements, vocalization, facial indicators, and posturing/guarding) and a physiological dimension (heart rate, blood pressure, respiratory rate, perspiration, and pallor). Each item is scored yes or no in this checklist. The PAIN was tested with 31 critically ill postoperative adult patients, most of them extubated. It showed acceptable criterion validity when the numbers of behavioral and physiological indicators observed were compared with the nurses' ratings of pain intensity (r=0.17 to 0.77, P<0.05). In a recent large study (Thunder Project II) conducted with 5957 critically ill adults, it was found that more behaviors (e.g., grimacing, rigidity, and vocalization) were exhibited by patients with procedural pain than without procedural pain.
      • Puntillo K.A.
      • Morris A.B.
      • Thompson C.L.
      • Stanik-Hutt J.
      • White C.
      Pain behaviors observed during six common procedures: results from Thunder Project II.
      The behavior observation tool used in that recent study was inspired from the PAIN and the Children's Hospital of Eastern Ontario Pain Scale (CHEOPS).
      • McGrath P.J.
      • Johnson G.I.
      • Goodman J.T.
      • et al.
      CHEOPS: a behavioral scale for rating postoperative pain in children.
      The BPS
      • Payen J.F.
      • Bru O.
      • Bosson J.L.
      • et al.
      Assessing pain in critically ill sedated patients by using a behavioral pain scale.
      includes three indicators (facial expression, movements of upper limbs, and compliance with the ventilator) scored from 1 to 4, for a possible total score ranging from 3 to 12. It was tested with three intensive care unit (ICU) mechanically ventilated and sedated adult samples: 30 trauma or postoperative patients,
      • Payen J.F.
      • Bru O.
      • Bosson J.L.
      • et al.
      Assessing pain in critically ill sedated patients by using a behavioral pain scale.
      30 medical patients,
      • Aïssaoui Y.
      • Zeggwagh A.A.
      • Zekraoui A.
      • Abidi K.
      • Abougal R.
      Validation of a behavioral pain scale in critically ill, sedated, and mechanically ventilated patients.
      and 44 medical or postoperative patients.
      • Young J.
      • Siffleet J.
      • Nikoletti S.
      • Shaw T.
      Use of a behavioral pain scale to assess pain in ventilated, unconscious and/or sedated patients.
      All patients were unconscious and unable to self-report. Interrater reliability was high in the first two studies,
      • Payen J.F.
      • Bru O.
      • Bosson J.L.
      • et al.
      Assessing pain in critically ill sedated patients by using a behavioral pain scale.
      • Aïssaoui Y.
      • Zeggwagh A.A.
      • Zekraoui A.
      • Abidi K.
      • Abougal R.
      Validation of a behavioral pain scale in critically ill, sedated, and mechanically ventilated patients.
      with a weighted κ-coefficient of 0.74 and an intraclass correlation coefficient of 0.95, respectively. However, interrater agreement varied from low to high (36%–91%) in the study of Young and colleagues,
      • Young J.
      • Siffleet J.
      • Nikoletti S.
      • Shaw T.
      Use of a behavioral pain scale to assess pain in ventilated, unconscious and/or sedated patients.
      where lower agreement was obtained during pain assessments of nociceptive procedures. Discriminant validity was supported in all three studies where higher BPS scores were obtained with nociceptive procedures compared with rest or non-nociceptive procedures. The inclusion of “compliance with the ventilator” was an interesting indicator to consider for pain assessment in intubated patients. Finally, the operational definition for movements of upper limbs could be confused with muscle tension.
      In summary, important efforts have been invested in research on the development of pain assessment tools in critically ill adults. Some of these tools showed limitations in their measurement scales or operational definition (PACU BPRS and BPS). The reliability and validity results for these tools were satisfactory. However, patient samples were not always representative of critically ill nonverbal adults (e.g., mechanically ventilated patients) and most sample sizes were small, limiting the generalizability of the findings. Also, criterion validity with the patients' self-reports of pain, which represents the most valid criterion, was not studied for some tools (PAIN and BPS). Except for the PAIN, for which no total intensity score can be calculated, sensitivity and specificity of the other existing tools were not documented.
      To overcome those limitations, a new tool, the Critical-Care Pain Observation Tool (CPOT), was initially developed in French and forward-backward translated into English. It includes four behavioral categories: 1) facial expression, 2) body movements, 3) muscle tension, and 4) compliance with the ventilator for intubated patients or vocalization for extubated patients.
      • Gélinas C.
      • Fillion L.
      • Puntillo K.A.
      • Viens C.
      • Fortier M.
      Validation of the Critical-Care Pain Observation Tool in adult patients.
      Items in each category are scored from 0 to 2 with a possible total score ranging from 0 to 8. Selection and operational definitions of the CPOT items were derived from the existing pain assessment instruments,
      • Mateo O.M.
      • Krenzischek D.A.
      A pilot study to assess the relationship between behavioral manifestations and self-report of pain in postanaesthesia care unit patients.
      • Puntillo K.A.
      • Miaskowski C.
      • Kehrle K.
      • et al.
      Relationship between behavioral and physiological indicators of pain, critical care patients' self-reports of pain, and opioid administration.
      • Payen J.F.
      • Bru O.
      • Bosson J.L.
      • et al.
      Assessing pain in critically ill sedated patients by using a behavioral pain scale.
      a chart review of 52 critically ill patients' medical files
      • Gélinas C.
      • Fortier M.
      • Viens C.
      • Fillion L.
      • Puntillo K.A.
      Pain assessment and management in critically-ill intubated patients: a retrospective study.
      and consultation with 48 critical care nurses and 12 physicians.
      • Gélinas C.
      • Viens C.
      • Fortier M.
      • Fillion L.
      Les indicateurs de douleur en soins critiques.
      The French and the English versions of the CPOT were tested with 105 cardiac surgery
      • Gélinas C.
      • Fillion L.
      • Puntillo K.A.
      • Viens C.
      • Fortier M.
      Validation of the Critical-Care Pain Observation Tool in adult patients.
      and 55 ICU adults with different diagnoses
      • Gélinas C.
      • Johnston C.
      Pain assessment in the mechanically ventilated adult: validation of the Critical-Care Pain Observation Tool and physiological indicators.
      (trauma, postoperative, and medical cases), respectively. It demonstrated moderate to high interrater reliability (French version: weighted κ 0.52–0.88; English version: intraclass correlation coefficients 0.80–0.93). Discriminant validity was supported with higher CPOT scores during a nociceptive procedure (turning) compared with rest or a non-nociceptive procedure. For criterion validity, the patients' self-reports of pain intensity were associated with the CPOT scores (e.g., correlation coefficients from 0.40 to 0.71, P0.05). Based on those results,
      • Gélinas C.
      • Fillion L.
      • Puntillo K.A.
      • Viens C.
      • Fortier M.
      Validation of the Critical-Care Pain Observation Tool in adult patients.
      • Gélinas C.
      • Johnston C.
      Pain assessment in the mechanically ventilated adult: validation of the Critical-Care Pain Observation Tool and physiological indicators.
      the CPOT seems to be a useful tool for critically ill nonverbal patients. Further steps in tool validation are still necessary, however, including measurement of sensitivity and specificity, which provide relevant and valuable information to clinicians.
      The analysis reported herein is derived from the larger study of Gélinas and colleagues
      • Gélinas C.
      • Fillion L.
      • Puntillo K.A.
      • Viens C.
      • Fortier M.
      Validation of the Critical-Care Pain Observation Tool in adult patients.
      and explores the psychometric qualities of the CPOT by evaluating its sensitivity and the specificity in intubated ICU adults after cardiac surgery. As a complementary objective, the association between the CPOT score and the patients' self-reports of pain intensity after extubation is also described.

      Methods

      Design, Sample, and Ethics

      A repeated measures design was chosen for this study, as described in a previous report.
      • Gélinas C.
      • Fillion L.
      • Puntillo K.A.
      • Viens C.
      • Fortier M.
      Validation of the Critical-Care Pain Observation Tool in adult patients.
      A convenience sample of 105 cardiac surgery ICU patients from a cardiology health center in Canada was recruited. Patients were considered for inclusion if they were 18 years old or older, admitted for cardiac surgery, understood French, were in the ICU after surgery, and were able to hear and see. Patients were excluded if any of the following conditions were present: admitted for a heart transplant or thoracic aortic aneurysm repair, received medical treatment for chronic pain, had an ejection fraction less than 25%, had pre-existing psychiatric or neurological problems, had a dependence on alcohol or drugs, received neuromuscular blockade postoperatively, and had complications after the surgery (e.g., hemorrhage, delirium, and death).
      The study was accepted by the human research committee of the health center. Recruitment was carried out the day before the surgery, and the study was explained to eligible patients. At this time, patients were taught how to use the pain intensity scales (Descriptive Pain Scale and Faces Pain Thermometer [FPT], see Measures section). The nurse research assistant explained to the patient the significance of both scales and the importance of rating their pain based on their own experience. Patients evaluated this teaching as essential to their care and supported that it should be provided to them before the surgery.

      Measures

      To allow determination of sensitivity and specificity of the CPOT, a gold standard criterion was necessary: the patients' self-reports of pain. We selected three verbal indicators: one for the patients while intubated and two for the patients after extubation. First, the yes or no response was obtained from intubated patients by head nodding for the presence or absence of pain. This pain criterion was selected because many intubated patients were unable to use pain intensity scales.
      Patients after extubation used two different pain intensity scales: Descriptive Pain Scale and FPT. Two pain scales were chosen because the FPT was a newly developed tool
      • Gélinas C.
      Le thermomètre d'intensité de douleur: un nouvel outil pour les patients en soins critiques.
      and the use of a known validated pain scale, the Descriptive Pain Scale, used with acutely ill and critically ill patients,
      • Mateo O.M.
      • Krenzischek D.A.
      A pilot study to assess the relationship between behavioral manifestations and self-report of pain in postanaesthesia care unit patients.
      • Choinière M.
      • Melzack R.
      • Girard N.
      • Rondeau J.
      • Paquin M.J.
      Comparisons between patients and nurses assessment of pain and medication efficacy in severe burn injuries.
      was essential to explore the validity of the FPT. Using a list of words of the Descriptive Pain Scale (none, mild, moderate, severe, unbearable), the patient was asked to choose the adjective that best described his/her current pain intensity. Finally, the FPT, developed for critically ill adults,
      • Gélinas C.
      Le thermomètre d'intensité de douleur: un nouvel outil pour les patients en soins critiques.
      consists of a thermometer graded from 0 to 10, including six faces adapted from the work of Prkachin
      • Prkachin K.M.
      The consistency of facial expressions of pain: a comparison across modalities.
      and other existing tools. The FPT was validated with the same 105 cardiac surgery ICU patients. The scale demonstrated good convergent validity (r=0.80 to 0.86, P<0.001 with the Descriptive Pain Scale described above) and discriminant validity (t=−5.10, P<0.001 comparing patient pain intensity at rest and during turning and yielding an association with a higher pain intensity score during turning). Content validity was also examined and patients positively evaluated its content and use.

      Procedure

      Six pain assessments with the CPOT, clustered into two testing periods, were completed in the patients' early postoperative period.
      • Gélinas C.
      • Fillion L.
      • Puntillo K.A.
      • Viens C.
      • Fortier M.
      Validation of the Critical-Care Pain Observation Tool in adult patients.
      Each testing period included three assessments: at rest before the procedure (pre-exposure), during a nociceptive procedure-turning (exposure), and 20 minutes after the procedure (postexposure). Patient turning represented a previously confirmed nociceptive procedure known to be painful in critically ill adults.
      • Puntillo K.A.
      • White C.
      • Morris A.B.
      • et al.
      Patients' perceptions and responses to procedural pain: results from Thunder Project II.
      The first testing period was completed while the patient was intubated and conscious. Consciousness was determined by a score of 2, 3, or 4 on the Ramsay Sedation Scale.
      • Ramsay M.
      • Savege T.
      • Simpson B.
      • Goodwin R.
      Controlled sedation with alphaxalone-alphadolone.
      Finally, the second testing period was completed after extubation, approximately five hours after the first testing period.
      For the two testing periods, patients were evaluated with the CPOT for one minute at rest (nonexposure), both before and after turning, and for the duration of the turning procedure (nociceptive exposure). This standardization of procedures was inspired by the work of Puntillo and colleagues.
      • Puntillo K.A.
      • White C.
      • Morris A.B.
      • et al.
      Patients' perceptions and responses to procedural pain: results from Thunder Project II.
      The principal investigator and one critical care nurse who was trained to use the CPOT evaluated the patients. Interrater reliability was supported with moderate to high weighted κ-coefficients (0.52–0.88) for the six pain assessments.
      • Gélinas C.
      • Fillion L.
      • Puntillo K.A.
      • Viens C.
      • Fortier M.
      Validation of the Critical-Care Pain Observation Tool in adult patients.
      On completion of the CPOT during the first testing period, intubated patients communicated the presence or absence of pain by head nodding (yes or no) to the question “Do you have pain?” During the second testing period, after completion of the CPOT, the extubated patients used the Descriptive Pain Scale and the FPT to grade their current pain intensity. To minimize bias, the CPOT evaluations were completed before asking the patient about their pain so that the CPOT raters were not influenced by the patient's self-report of pain.

      Data Analysis

      Data collected in the first testing period were used in a Receiver-Operating Characteristic (ROC) curve analysis to evaluate the ability of the CPOT to detect pain of intubated patients and to derive the threshold that maximizes both the sensitivity and specificity simultaneously.
      • Zweig M.H.
      • Campbell G.
      Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine.
      Sensitivity refers to the true-positive rate. That is, a test or a tool is sensitive in detecting a disorder (i.e., pain) when it is actually present (i.e., as reported by patients). Specificity refers to the true-negative rate. In other words, the test should be negative in patients who reported having no pain. The ROC curve is a plot of sensitivity (the true-positive fraction) against 1 minus specificity (the false-positive fraction) pairs resulting from different values of the CPOT (0–8). Perfect discrimination has a plot that passes through the upper left corner of the plot where the true-positive fraction is 100% and the false-positive fraction is 0. The closer the plot is to the upper left corner of the curve, the higher the overall predictive power of the criterion. The area under the curve (AUC) provides a measure (on a scale from 0.5 to 1 where 0.5 represents discrimination no better than chance and 1 perfect discrimination) of the CPOT's ability to distinguish between patients who reported having pain (yes) and those who reported not having pain (no). It was tested if the area under the ROC curve was greater than 50% using an alpha level of significance of 0.05. Using the threshold suggested by the ROC curve, sensitivity and specificity of the CPOT were calculated.
      Likelihood ratios (LRs), which are alternative statistics for summarizing diagnostic accuracy, were also calculated to evaluate the ability of the threshold CPOT value to “rule in” (LR positive [LR+]) or “rule out” (LR negative [LR]) the presence or the absence of pain reported by intubated patients. An LR+ describes how much more likely a positive test result is to be found in intubated patients who reported having pain compared with those who reported having no pain. A value greater than 1 is expected for LR+. An LR describes how many times more likely a negative test result is to be found in intubated patients who reported having no pain compared with those who reported having pain. A value less than 1 is expected for LR.
      • Fischer J.E.
      • Bachmann L.M.
      • Jaeschke R.
      A readers' guide to the interpretation of diagnostic test properties: clinical example of sepsis.
      Finally, the association between the CPOT's threshold value and the extubated patients' self-reports of pain intensity obtained at the second testing period was examined through use of Chi-square tests. This association was tested using the Descriptive Pain Scale and the FPT, separately. To perform the test, patients' self-reports of pain intensity on the FPT were clustered into four categories:
      • Daut R.L.
      • Cleeland C.S.
      • Flanery R.C.
      Development of the Wisconsin Brief Pain Questionnaire to assess pain in cancer and other diseases.
      1) no pain: score of 0; 2) mild pain: score from 1 to 3; 3) moderate pain: score from 4 to 6; and 4) severe pain: score from 7 to 10.

      Results

      Sample

      As noted in an earlier publication,
      • Gélinas C.
      • Fillion L.
      • Puntillo K.A.
      • Viens C.
      • Fortier M.
      Validation of the Critical-Care Pain Observation Tool in adult patients.
      a total of 131 patients were approached for consent the day before surgery, and 117 (89%) agreed to participate in the study. Reasons for refusal were 1) anxious about the surgery (n=9), 2) not interested (n=3), 3) undecided (n=1), and 4) bad experience with research (n=1). During the course of the study, eight patients were excluded due to postoperative complications (hemorrhage, delirium, and death); three due to surgery cancellation; and one due to extubation right after surgery. The final sample size was 105 patients enrolled over a three-month time period. Eighty-three were males (79%) and 22 were females (21%), with a mean age of 60 years (standard deviation=8). Most patients underwent coronary artery bypass graft (CABG, 79%), some valvular surgery (10.5%), both CABG and valvular surgery (8.6%), or interauricular/interventricular communication repair (1.9%). All patients had sternal incisions. Patients were receiving continuous infusions of fentanyl when they were admitted to the ICU from surgery. The fentanyl decreased in amount from 65.1±29.0 μg/hour in intubated conscious patients to 50.7±31.1 μg/hour after extubation. Intravenous bolus of fentanyl was not part of the pain protocol and was only available on an individual basis (physician's prescription). Only four patients received a bolus of fentanyl before turning.
      Data collection was completed on 99 of the 105 awake intubated patients. The remaining six patients were extubated before the completion of this testing period. Finally, all 105 patients were assessed once extubated.

      Sensitivity and Specificity of the CPOT

      More than 50% of intubated patients reported pain while at rest. Being in pain was observed at pre- and postexposure. This percentage was higher (80%) during nociceptive exposure (see Table 1). ROC curve analysis was performed at these three assessments (see Fig. 1, Fig. 2, Fig. 3). During and after the nociceptive exposure, AUC indicated intermediate to good discriminative properties (i.e., AUC of 0.75–0.85). During nociceptive exposure, the threshold associated with maximization of the sums of sensitivity and specificity was found to be a score greater than 2 on the CPOT, whereas for the nonexposure periods (pre- and postexposure), the threshold was found to be a score greater than 1 on the CPOT. Properties of the CPOT for the detection of pain in intubated patients are presented in Table 2. At these thresholds, sensitivity was higher than specificity during the nociceptive exposure. In contrast, specificity was higher than sensitivity during nonexposure (pre- and postexposure). At pre-exposure and during nociceptive exposure, a patient who reported to be in pain had a three to four times greater chance (LR+) for a positive score on the CPOT (>2) than a patient who reported no pain. The LR+ was much higher at postexposure. Finally, the accuracy rate of the CPOT to detect presence or absence of pain in intubated patients was good during nociceptive exposure and at postexposure, but was lower at pre-exposure. In summary, the CPOT was effective in showing good detection properties, particularly during and after the nociceptive exposure, when pain could be higher.
      Table 1Intubated Patients' Self-Report of Pain (Yes or No) at the First Testing Period
      AssessmentYes, Presence of Pain n (%)No, Absence of Pain n (%)
      Pre-exposure53 (56.4)41 (43.6)
      Nociceptive exposure79 (81.4)18 (18.6)
      Postexposure54 (58.1)39 (41.9)
      Because of intermittent drowsiness postoperatively, five patients at pre-exposure, two during nociceptive exposure, and six at postexposure were unable to give their self-reports of pain by head nodding.
      Figure thumbnail gr1
      Fig. 1ROC curve at pre-exposure in intubated patients.
      Figure thumbnail gr2
      Fig. 2ROC curve during nociceptive exposure in intubated patients.
      Figure thumbnail gr3
      Fig. 3ROC curve at postexposure in intubated patients.
      Table 2Properties of the CPOT for the Detection of Pain in Intubated Patients at the First Testing Period
      PeriodPropertiesValue95% CI
      Pre-exposureSensitivity25/53=47.2%[34.4%, 60.3%]
      Specificity34/41=82.9%[68.7%, 91.5%]
      LR+2.763[1.328, 5.746]
      LR0.637[0.477, 0.851]
      Accuracy59/94=62.8%[52.7%, 71.9%]
      During nociceptive exposureSensitivity68/79=86.1%[76.8%, 92.0%]
      Specificity14/18=77.8%[54.8%, 91.0%]
      LR+3.873[1.625, 9.234]
      LR0.179[0.098, 0.327]
      Accuracy82/97=84.5%[76.0%, 90.4%]
      PostexposureSensitivity34/54=63.0%[49.6%, 74.6%]
      Specificity38/39=97.4%[86.8%, 99.5%]
      LR+24.556[3.510, 171.811]
      LR0.380[0.267, 0.540]
      Accuracy72/93=77.4%[67.9%, 84.7%]
      CI=confidence interval.

      Association Between the Extubated Patients' Self-Reports of Pain Intensity and the CPOT Threshold Value

      Extubated patients' self-reports of pain intensity were associated with the CPOT threshold value (a score greater than 1 on the CPOT at rest (pre- and postexposure) and a score greater than 2 on the CPOT during nociceptive exposure) previously determined while the patients were intubated (see Table 3). Chi-squared tests were significant for the three assessments (pre-exposure, nociceptive exposure, and postexposure) for both pain intensity scales. It was observed that the CPOT was able to adequately classify most patients with severe or unbearable pain. At pre-exposure, seven patients with severe or unbearable pain out of 10 (70%) on the Descriptive Pain Scale and seven out of 11 (64%) with severe pain on the FPT obtained a score >1 on the CPOT. During the nociceptive exposure, 21 patients with severe or unbearable pain out of 25 (84%) on the Descriptive Pain Scale and 17 out of 20 (85%) with severe pain on the FPT obtained a score >2 on the CPOT. Finally, at postexposure, results were weaker. Four patients with severe pain out of six (67%) on the Descriptive Pain Scale and only one patient out of five (20%) with severe pain on the FPT obtained a score >1 on the CPOT.
      Table 3Association Between the Extubated Patients' Self-Report of Pain Intensity and CPOT Threshold Value at the Second Testing Period
      PeriodPain IntensityLevelsCPOTχ2
      Negative
      A negative CPOT score is below the threshold value (≤1 pre- and postexposure, ≤2 during nociceptive exposure).
      Positive
      A positive CPOT score is above the threshold value (>1 pre- and postexposure, >2 during nociceptive exposure).
      n/N%n/N%
      Pre-exposureDescriptive Pain Scale (n=104)None7/897.90/150.034.2
      P≤0.001.
      Mild29/8932.60/150.0
      Moderate50/8956.28/1553.3
      Severe3/893.44/1526.7
      Unbearable0/890.03/1520.0
      FPT (n=101)None (0)8/899.00/120.033.6
      P≤0.001.
      Mild (1–3)31/8934.80/120.0
      Moderate (4–6)46/8951.75/1241.7
      Severe (7–10)4/894.57/1258.3
      During nociceptive exposureDescriptive Pain Scale (n=104)None6/4214.30/620.022.9
      P≤0.001.
      Mild13/4231.06/629.7
      Moderate19/4245.235/6256.5
      Severe4/429.514/6222.6
      Unbearable0/420.07/6211.3
      FPT (n=102)None (0)6/4114.60/610.021.6
      P≤0.001.
      Mild (1–3)14/4134.17/6111.5
      Moderate (4–6)18/4143.937/6160.7
      Severe (7–10)3/417.317/6127.9
      PostexposureDescriptive Pain Scale (n=104)None10/8212.20/220.016.5
      P≤0.001.
      Mild43/8252.45/2222.7
      Moderate27/8232.913/2259.1
      Severe2/822.44/2218.2
      FPT (n=99)None (0)9/8011.30/190.016.9
      P≤0.001.
      Mild (1–3)47/8058.84/1921.1
      Moderate (4–6)20/8025.014/1973.7
      Severe (7–10)4/805.01/195.3
      Because of intermittent drowsiness postoperatively, some patients were unable to give their self-reports of pain by head nodding.
      a A negative CPOT score is below the threshold value (≤1 pre- and postexposure, ≤2 during nociceptive exposure).
      b A positive CPOT score is above the threshold value (>1 pre- and postexposure, >2 during nociceptive exposure).
      c P0.001.

      Discussion

      This report described additional psychometric qualities (i.e., sensitivity and specificity) from a larger validation study of the CPOT, a behavioral pain assessment tool developed for clinical use in critically ill intubated adults.
      • Gélinas C.
      • Fillion L.
      • Puntillo K.A.
      • Viens C.
      • Fortier M.
      Validation of the Critical-Care Pain Observation Tool in adult patients.
      The specificity of the CPOT was high for all assessments of intubated and conscious patients: the higher the specificity of a test or a tool, the lower the false-positive cases.
      • Zweig M.H.
      • Campbell G.
      Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine.
      • Fischer J.E.
      • Bachmann L.M.
      • Jaeschke R.
      A readers' guide to the interpretation of diagnostic test properties: clinical example of sepsis.
      In other words, high specificity would help to “rule in” patients reporting having pain.
      • Boyko E.J.
      Ruling out or ruling in disease with the most sensitive or specific diagnostic test: short cut or wrong turn?.
      Thus, the risks of administering an analgesic to a patient who does not need it and the potential adverse effects that could result from the medication would be avoided.
      Sensitivity was high during the nociceptive exposure, moderate at postexposure, and low at pre-exposure. A possible explanation would be that, when patients are not exposed to a nociceptive procedure, some of them do not show or are able to control their behaviors even if they feel pain. This might be different when they are exposed to a nociceptive procedure. That is, they might be unable to control reflexive pain behaviors, which would explain the higher sensitivity during turning (nociceptive exposure). Patients with a positive CPOT (>2) score indicating the presence of pain during a nociceptive procedure would benefit from receiving an analgesic. Indeed, administering an analgesic before a nociceptive procedure, such as turning, may not be harmful and could prevent procedural pain.
      • Agency for Health Care Policy and Research (AHCPR)
      Acute pain management: operative or medical procedures and trauma, part 1.
      • Herr K.
      • Coyne P.J.
      • Key T.
      • et al.
      Pain assessment in the nonverbal patient: position statement with clinical practice recommendations.
      All patients in this study received continuous infusions of fentanyl, but rarely (n=4) received an intravenous bolus before turning; this precluded exploration of the impact of this intervention on the CPOT findings.
      The positive CPOT threshold value differed according to the exposure status. It was lower when not exposed (>1) and higher during exposure (>2). These results may be explained in two ways. First, intubated patients may react more intensely to pain when they are exposed to a nociceptive procedure (turning) than while they are at rest. This is not surprising because turning may cause additional pain to these patients. In a previous study, turning was identified as the most painful of six procedures performed on acutely and critically ill adults.
      • Puntillo K.A.
      • White C.
      • Morris A.B.
      • et al.
      Patients' perceptions and responses to procedural pain: results from Thunder Project II.
      Second, the nurse may perceive the presence of more behaviors during patient turning if she knows the procedure is painful. So, if the nurse thinks her patient can tolerate turning and observes a high score on the CPOT, she may give an analgesic bolus before the next turning procedure and also right after the procedure.
      For extubated patients, an association was found between patients' self-reports of pain intensity and the CPOT. The more severe the patients' pain, the more likely a positive result on the CPOT was found. In a meta-analysis, Labus and colleagues
      • Labus J.S.
      • Keefe F.J.
      • Jensen M.P.
      Self-reports of pain intensity and direct observations of pain behavior: when are they correlated?.
      found that pain behaviors were positively and moderately associated with patients' self-reports of pain intensity. Also, in two previous studies of Gélinas and colleagues,
      • Gélinas C.
      • Fillion L.
      • Puntillo K.A.
      • Viens C.
      • Fortier M.
      Validation of the Critical-Care Pain Observation Tool in adult patients.
      • Gélinas C.
      • Johnston C.
      Pain assessment in the mechanically ventilated adult: validation of the Critical-Care Pain Observation Tool and physiological indicators.
      the patients' self-reports of pain intensity were moderately to highly correlated with the CPOT scores (r=0.40–0.71, P<0.05). Even if pain behaviors are known to be related to the patients' self-reports of pain intensity, it must be emphasized that these two indicators of pain are different in nature (behavioral and sensory components of pain) and may be complementary. Thus, it is not surprising to find that some patients with moderate or mild pain obtained a positive score (>1 at pre- and postexposure and >2 during nociceptive exposure) on the CPOT (false-positive cases), and that other patients with severe pain obtained a negative score on the CPOT (false-negative cases). As highlighted in the clinical recommendations of the American Society for Pain Management Nursing,
      • Herr K.
      • Coyne P.J.
      • Key T.
      • et al.
      Pain assessment in the nonverbal patient: position statement with clinical practice recommendations.
      the use of behaviors seems to be more relevant for the detection of the presence of pain than for the interpretation of pain levels (mild, moderate, and severe). In fact, the CPOT was more effective in adequately classifying patients with severe pain than those with mild or moderate pain. Finally, it must be highlighted that patients were intubated during the first testing period and were extubated during the second testing period. Thus, the patients' clinical conditions had changed as they recovered postoperatively. That CPOT thresholds may differ between intubated and extubated patients is an interesting possibility that warrants further exploration.
      The potential limitations of this study must be considered. The postoperative pain experience may be different for patients with other conditions, which could alter study results. Also, the gold standard (presence or absence of pain: yes or no) used for evaluation of sensitivity and specificity of the CPOT in intubated patients was limited because it did not allow us to describe the degree of the patients' pain intensity. Moreover, postoperative drowsiness led to missing data for some patients. Finally, potential bias of the investigator and her colleague are possible. More raters should be used in subsequent evaluations of the CPOT.
      Although the patient's self-report represents the most valid measure for pain, it may be impossible to use in many critically ill patients. As already noted by Anand and Craig,
      • Anand K.J.S.
      • Craig K.D.
      New perspectives on the definition of pain.
      and as recommended in clinical practice guidelines
      • Herr K.
      • Coyne P.J.
      • Key T.
      • et al.
      Pain assessment in the nonverbal patient: position statement with clinical practice recommendations.
      • Jacobi J.
      • Fraser G.L.
      • Coursin D.B.
      • et al.
      Clinical practice guidelines for the sustained use of sedatives and analgesics in the critically ill adult.
      for critically ill nonverbal adults, pain behaviors are valuable forms of self-report and should be considered as alternative measures of pain. The change in these behaviors after analgesic therapy should also be assessed. Thus, the use of the CPOT may expedite pain management in patients with difficulties in communication.
      In future work, the sensitivity and the specificity of the CPOT need to be studied in different critical care populations and associated with a variety of nociceptive procedures. Inclusion of other behavioral or physiological indicators in the tool could also be considered for testing.

      Conclusions

      The specificity of the CPOT was high across all conditions (no exposure and nociceptive exposure), but sensitivity was high only during nociceptive exposure. Sensitivity and specificity of behavioral indicators need to be further explored in different critically ill populations and various situations or procedures. Development of reliable and valid pain assessment tools to be used for patients who are not able to self-report is an essential first step toward better pain management for this vulnerable group of critically ill patients. The CPOT is a valid tool and various psychometric qualities have been examined. Available in English and French, it is already being used for research or clinical purposes in North America and requests to translate the tool in other languages were recently made.

      References

        • Loeser J.D.
        • Cousins M.J.
        Contemporary pain management.
        Med J Aust. 1990; 153 (216): 208-212
        • McCaffery M.
        Nursing management of the patient with pain.
        2nd ed. Lippincott, Philadelphia1979
        • Rotondi A.J.
        • Chelluri L.
        • Sirio C.
        • et al.
        Patients' recollections of stressful experiences while receiving prolonged mechanical ventilation in an intensive care unit.
        Crit Care Med. 2002; 30: 746-752
        • Gélinas C.
        Management of pain in cardiac surgery ICU patients: have we improved over time?.
        Intensive Crit Care Nurs. 2007; 23: 298-303
        • Puntillo K.A.
        • White C.
        • Morris A.B.
        • et al.
        Patients' perceptions and responses to procedural pain: results from Thunder Project II.
        Am J Crit Care. 2001; 10: 238-251
        • Stanik-Hutt J.
        • Soeken K.L.
        • Fontaine D.K.
        • Gift A.G.
        Pain experiences of traumatically injured patients in a critical care setting.
        Am J Crit Care. 2001; 10: 252-259
        • Hamill-Ruth R.J.
        • Marohn L.
        Evaluation of pain in the critically ill patient.
        Crit Care Clin. 1999; 15: 35-53
        • Shannon K.
        • Bucknall T.
        Pain assessment in critical care: what have we learnt from research.
        Intensive Crit Care Nurs. 2003; 19: 154-162
        • Agency for Health Care Policy and Research (AHCPR)
        Acute pain management: operative or medical procedures and trauma, part 1.
        Clin Pharm. 1992; 11: 309-331
        • Anand K.J.S.
        • Craig K.D.
        New perspectives on the definition of pain.
        Pain. 1996; 67: 3-6
        • Herr K.
        • Coyne P.J.
        • Key T.
        • et al.
        Pain assessment in the nonverbal patient: position statement with clinical practice recommendations.
        Pain Manag Nurs. 2006; 7: 44-52
        • Mateo O.M.
        • Krenzischek D.A.
        A pilot study to assess the relationship between behavioral manifestations and self-report of pain in postanaesthesia care unit patients.
        J Post Anesth Nurs. 1992; 7: 15-21
        • Puntillo K.A.
        • Miaskowski C.
        • Kehrle K.
        • et al.
        Relationship between behavioral and physiological indicators of pain, critical care patients' self-reports of pain, and opioid administration.
        Crit Care Med. 1997; 25: 1159-1166
        • Payen J.F.
        • Bru O.
        • Bosson J.L.
        • et al.
        Assessing pain in critically ill sedated patients by using a behavioral pain scale.
        Crit Care Med. 2001; 29: 2258-2263
        • Webb M.R.
        • Kennedy M.G.
        Behavioral responses and self-reported pain in postoperative patients.
        J Post Anesth Nurs. 1994; 9: 91-95
        • Feinstein A.R.
        Clinimetrics.
        Yale University Press, New Haven1987
        • Puntillo K.A.
        • Morris A.B.
        • Thompson C.L.
        • Stanik-Hutt J.
        • White C.
        Pain behaviors observed during six common procedures: results from Thunder Project II.
        Crit Care Med. 2004; 32: 421-427
        • McGrath P.J.
        • Johnson G.I.
        • Goodman J.T.
        • et al.
        CHEOPS: a behavioral scale for rating postoperative pain in children.
        Adv Pain Res Ther. 1985; 9: 395-402
        • Aïssaoui Y.
        • Zeggwagh A.A.
        • Zekraoui A.
        • Abidi K.
        • Abougal R.
        Validation of a behavioral pain scale in critically ill, sedated, and mechanically ventilated patients.
        Anesth Analg. 2005; 101: 1470-1476
        • Young J.
        • Siffleet J.
        • Nikoletti S.
        • Shaw T.
        Use of a behavioral pain scale to assess pain in ventilated, unconscious and/or sedated patients.
        Intensive Crit Care Nurs. 2006; 22: 32-39
        • Gélinas C.
        • Fillion L.
        • Puntillo K.A.
        • Viens C.
        • Fortier M.
        Validation of the Critical-Care Pain Observation Tool in adult patients.
        Am J Crit Care. 2006; 15: 420-427
        • Gélinas C.
        • Fortier M.
        • Viens C.
        • Fillion L.
        • Puntillo K.A.
        Pain assessment and management in critically-ill intubated patients: a retrospective study.
        Am J Crit Care. 2004; 13: 126-135
        • Gélinas C.
        • Viens C.
        • Fortier M.
        • Fillion L.
        Les indicateurs de douleur en soins critiques.
        Perspect Infirmière. 2005; 2: 12-22
        • Gélinas C.
        • Johnston C.
        Pain assessment in the mechanically ventilated adult: validation of the Critical-Care Pain Observation Tool and physiological indicators.
        Clin J Pain. 2007; 23: 497-505
        • Gélinas C.
        Le thermomètre d'intensité de douleur: un nouvel outil pour les patients en soins critiques.
        Perspect Infirmière. 2007; 4: 12-20
        • Choinière M.
        • Melzack R.
        • Girard N.
        • Rondeau J.
        • Paquin M.J.
        Comparisons between patients and nurses assessment of pain and medication efficacy in severe burn injuries.
        Pain. 1990; 40: 143-152
        • Prkachin K.M.
        The consistency of facial expressions of pain: a comparison across modalities.
        Pain. 1992; 51: 297-306
        • Ramsay M.
        • Savege T.
        • Simpson B.
        • Goodwin R.
        Controlled sedation with alphaxalone-alphadolone.
        BMJ. 1974; 2: 656-659
        • Zweig M.H.
        • Campbell G.
        Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine.
        Clin Chem. 1993; 39: 561-577
        • Fischer J.E.
        • Bachmann L.M.
        • Jaeschke R.
        A readers' guide to the interpretation of diagnostic test properties: clinical example of sepsis.
        Intensive Care Med. 2003; 29: 1043-1051
        • Daut R.L.
        • Cleeland C.S.
        • Flanery R.C.
        Development of the Wisconsin Brief Pain Questionnaire to assess pain in cancer and other diseases.
        Pain. 1983; 17: 197-210
        • Boyko E.J.
        Ruling out or ruling in disease with the most sensitive or specific diagnostic test: short cut or wrong turn?.
        Med Decis Making. 1994; 14: 175-179
        • Labus J.S.
        • Keefe F.J.
        • Jensen M.P.
        Self-reports of pain intensity and direct observations of pain behavior: when are they correlated?.
        Pain. 2003; 102: 109-124
        • Jacobi J.
        • Fraser G.L.
        • Coursin D.B.
        • et al.
        Clinical practice guidelines for the sustained use of sedatives and analgesics in the critically ill adult.
        Crit Care Med. 2002; 30: 119-141