Advertisement

Fidelity and Feasibility of a Brief Emergency Department Intervention to Empower Adults With Serious Illness to Initiate Advance Care Planning Conversations

  • Richard E. Leiter
    Correspondence
    Address correspondence to: Richard E. Leiter, MD, MA, Department of Psychosocial Oncology and Palliative Care, Dana-Farber Cancer Institute, 450 Brookline Ave, LW-603, Boston, MA 02215, USA.
    Affiliations
    Department of Psychosocial Oncology and Palliative Care, Dana-Farber Cancer Institute, Boston, Massachusetts, USA

    Division of Palliative Medicine, Department of Medicine, Brigham and Women's Hospital, Boston, Massachusetts, USA
    Search for articles by this author
  • Miryam Yusufov
    Affiliations
    Department of Psychosocial Oncology and Palliative Care, Dana-Farber Cancer Institute, Boston, Massachusetts, USA

    Department of Psychiatry, Brigham and Women's Hospital, Boston, Massachusetts, USA
    Search for articles by this author
  • Mohammad Adrian Hasdianda
    Affiliations
    Department of Emergency Medicine, Brigham and Women's Hospital, Boston, Massachusetts, USA

    Department of Emergency Medicine, Harvard Medical School, Boston, Massachusetts, USA
    Search for articles by this author
  • Lauren A. Fellion
    Affiliations
    Department of Emergency Medicine, Brigham and Women's Hospital, Boston, Massachusetts, USA

    Department of Emergency Medicine, Harvard Medical School, Boston, Massachusetts, USA
    Search for articles by this author
  • Audrey C. Reust
    Affiliations
    Department of Emergency Medicine, Brigham and Women's Hospital, Boston, Massachusetts, USA

    Department of Emergency Medicine, Harvard Medical School, Boston, Massachusetts, USA
    Search for articles by this author
  • Susan D. Block
    Affiliations
    Department of Psychosocial Oncology and Palliative Care, Dana-Farber Cancer Institute, Boston, Massachusetts, USA

    Department of Emergency Medicine, Harvard Medical School, Boston, Massachusetts, USA

    Serious Illness Care Program, Ariadne Labs, Boston, Massachusetts, USA

    Department of Psychiatry, Brigham and Women's Hospital, Boston, Massachusetts, USA
    Search for articles by this author
  • James A. Tulsky
    Affiliations
    Department of Psychosocial Oncology and Palliative Care, Dana-Farber Cancer Institute, Boston, Massachusetts, USA

    Division of Palliative Medicine, Department of Medicine, Brigham and Women's Hospital, Boston, Massachusetts, USA
    Search for articles by this author
  • Kei Ouchi
    Affiliations
    Department of Psychosocial Oncology and Palliative Care, Dana-Farber Cancer Institute, Boston, Massachusetts, USA

    Department of Emergency Medicine, Brigham and Women's Hospital, Boston, Massachusetts, USA

    Department of Emergency Medicine, Harvard Medical School, Boston, Massachusetts, USA

    Serious Illness Care Program, Ariadne Labs, Boston, Massachusetts, USA
    Search for articles by this author
Open ArchivePublished:September 14, 2018DOI:https://doi.org/10.1016/j.jpainsymman.2018.09.003

      Abstract

      Context

      Emergency department (ED) visits provide opportunities to empower patients to discuss advance care planning with their outpatient clinicians, but systematically developed, feasible interventions do not currently exist. Brief negotiated interview (BNI) interventions, which allow ED clinicians to efficiently motivate patients, have potential to meet this need.

      Objectives

      We developed a BNI ED intervention to empower older adults with life-limiting illness to formulate and communicate medical care goals to their primary outpatient clinicians. This study assessed the fidelity and feasibility of this intervention in a high-volume ED.

      Methods

      We enrolled adult patients with serious illnesses (advanced cancer, congestive heart failure, chronic obstructive pulmonary disease, chronic kidney disease on dialysis, predicted survival <12 months) in an urban, tertiary care academic medical center ED. All participants received the BNI intervention. We video-recorded the encounters. Two reviewers assessed the recordings for intervention fidelity based on adherence to the BNI steps (Part I) and communication skills (Part II).

      Results

      We reviewed 46 video recordings. The mean total adherence score was 21.07/27 (SD 3.68) or 78.04%. The Part I mean adherence score was 12.07/15 (SD 2.07) or 80.47%. The Part II mean adherence score was 9.0/12 (SD 2.51) or 75%. The majority (75.6%) of recordings met the prespecified threshold for high intervention fidelity.

      Conclusion

      ED clinicians can deliver a BNI intervention to increase advance care planning conversations with high fidelity. Future research is needed to study the intervention's efficacy in a wider patient population.

      Key Words

      Introduction

      Seventy-five percent of older adults with life-limiting illnesses visit the emergency department (ED) in the last six months of life.
      • Smith A.K.
      • McCarthy E.
      • Weber E.
      • et al.
      Half of older Americans seen in emergency department in last month of life; most admitted to hospital, and many die there.
      Most have priorities other than to live as long as possible,
      • Steinhauser K.E.
      • Christakis N.A.
      • Clipp E.C.
      • McNeilly M.
      • McIntyre L.
      • Tulsky J.A.
      Factors considered important at the end of life by patients, family, physicians, and other care providers.
      yet up to 79% of older adults in the ED have not completed advance directives.
      • Oulton J.
      • Rhodes S.M.
      • Howe C.
      • Fain M.J.
      • Mohler M.J.
      Advance directives for older adults in the emergency department: a systematic review.
      ED visits that do not require in-the-moment decision making (e.g., a code status conversation) represent opportunities to empower seriously ill patients to formulate and communicate their goals for future medical care through advance care planning (ACP). ACP conversations are associated with increased hospice use, fewer hospitalizations, and lower rates of in-hospital death and intensive life-sustaining treatments at the end of life.
      • Lakin J.R.
      • Block S.D.
      • Billings J.A.
      • et al.
      Improving communication about serious illness in primary care.
      • Bernacki R.E.
      • Block S.D.
      Communication about serious illness care goals: a review and synthesis of best practices.
      • Shen M.J.
      • Prigerson H.G.
      • Paulk E.
      • et al.
      Impact of end-of-life discussions on the reduction of Latino/non-Latino disparities in do-not-resuscitate order completion.
      • Khandelwal N.
      • Kross E.K.
      • Engelberg R.A.
      • et al.
      Estimating the effect of palliative care interventions and advance care planning on ICU utilization.
      However, we currently lack feasible, evidence-based methods to facilitate ACP in the ED.
      ED clinicians are interested in engaging seriously ill adults in ACP conversations
      • Stone S.C.
      • Mohanty S.
      • Grudzen C.R.
      • et al.
      Emergency medicine physicians' perspectives of providing palliative care in an emergency department.
      but multiple barriers prevent them from doing so. ED clinicians perceive these conversations to be time intensive,
      • Stone S.C.
      • Mohanty S.
      • Grudzen C.R.
      • et al.
      Emergency medicine physicians' perspectives of providing palliative care in an emergency department.
      • Wright R.J.
      • Lowton K.
      • Robert G.
      • Grudzen C.R.
      • Grocott P.
      Emergency department staff priorities for improving palliative care provision for older people: a qualitative study.
      and they often lack training in serious illness communication.
      • Smith A.K.
      • Fisher J.
      • Schonberg M.A.
      • et al.
      Am I doing the right thing? Provider perspectives on improving palliative care in the emergency department.
      The hectic ED environment also makes it difficult for patients and clinicians to have sensitive and difficult discussions.
      To overcome known barriers and address these ACP needs, we have developed, refined, and tested a brief negotiated interview (BNI) ED intervention to empower older adults with life-limiting illness to initiate ACP conversations with their outpatient clinicians. BNI interventions are theoretically grounded in motivational interviewing
      • Rollnick S.
      • Miller W.R.
      What is motivational interviewing?.
      and are designed specifically for the ED. They allow clinicians to respectfully elicit patient perspectives and provide information and resources to motivate patients to take better control of their health care decisions, which promotes improved health outcomes and experiences.
      • Hibbard J.H.
      • Greene J.
      What the evidence shows about patient activation: better health outcomes and care experiences; fewer data on costs.
      BNI interventions have effectively reduced future substance misuse in patients with alcohol and substance use disorders.
      • D'Onofrio G.
      • Fiellin D.A.
      • Pantalon M.V.
      • et al.
      A brief intervention reduces hazardous and harmful drinking in emergency department patients.
      • Bernstein E.
      • Edwards E.
      • Dorfman D.
      • et al.
      Screening and brief intervention to reduce Marijuana use among youth and young adults in a pediatric emergency department.
      • Bernstein J.
      • Bernstein E.
      • Tassiopoulos K.
      • et al.
      Brief motivational intervention at a clinic visit reduces cocaine and heroin use.
      • Bruguera P.
      • Barrio P.
      • Oliveras C.
      • et al.
      Effectiveness of a specialized brief intervention for at risk drinkers in an emergency department: short term results of a randomized controlled trial.
      • Barata I.A.
      • Shandro J.R.
      • Montgomery M.
      • et al.
      Effectiveness of SBIRT for alcohol use disorders in the emergency department: a systematic review.
      Only recently, though, have BNI interventions been applied to ACP.
      • Fried T.R.
      • Leung S.L.
      • Blakley L.A.
      • Martino S.
      Development and pilot testing of a motivational interview for engagement in advance care planning.
      The primary goal of this pilot study was to assess whether trained ED clinicians can administer a structured BNI intervention to facilitate ACP conversations with patients' primary outpatient clinicians with high fidelity, as measured by adherence to steps of the BNI and serious illness communication skills. A secondary goal was to assess whether this intervention is feasible to deliver in a high-volume ED in an urban, tertiary care academic medical center, as measured by the percent of eligible patients enrolled and time spent delivering the intervention.

      Methods

      Intervention Development and Refinement

      A detailed description of the intervention's development will be published elsewhere. In brief, we conducted a systematic development process to design the BNI intervention using rapid qualitative inquiry.
      • Beebe J.
      Rapid qualitative inquiry: A field guide to team-based assessment.
      We created a prototype of our intervention by adapting a previously described BNI intervention established for alcohol dependence
      • Bernstein E.
      • Bernstein J.
      • Feldman J.
      • et al.
      An evidence based alcohol screening, brief intervention and referral to treatment (SBIRT) curriculum for emergency department (ED) providers improves skills and utilization.
      to fit the needs of patients with serious illness using literature review and an expert panel. Our expert panel consisted of palliative care physicians/researchers, a psychiatrist, and ED physicians/BNI researchers who have internationally recognized expertise in the development and implementation of interventions to improve clinician communication with seriously ill patients across medical settings, as well as the BNI method.
      • Lakin J.R.
      • Block S.D.
      • Billings J.A.
      • et al.
      Improving communication about serious illness in primary care.
      • Bernstein E.
      • Bernstein J.
      • Feldman J.
      • et al.
      An evidence based alcohol screening, brief intervention and referral to treatment (SBIRT) curriculum for emergency department (ED) providers improves skills and utilization.
      • Back A.L.
      • Arnold R.M.
      • Tulsky J.A.
      • Baile W.F.
      • Fryer-Edwards K.A.
      Teaching communication skills to medical oncology fellows.
      • Pollak K.I.
      • Back A.L.
      • Tulsky J.A.
      Disseminating effective clinician communication techniques: engaging clinicians to want to learn how to engage patients.
      • Back A.L.
      • Arnold R.M.
      • Baile W.F.
      • et al.
      Faculty development to change the paradigm of communication skills teaching in oncology.
      • Cooper Z.
      • Koritsanszky L.A.
      • Cauley C.E.
      • et al.
      Recommendations for best communication practices to facilitate goal-concordant care for seriously ill older patients with emergency surgical conditions.
      • Lakin J.R.
      • Koritsanszky L.A.
      • Cunningham R.
      • et al.
      A systematic intervention to improve serious illness communication in primary care.
      • Bernacki R.
      • Hutchings M.
      • Vick J.
      • et al.
      Development of the serious illness care program: a randomised controlled trial of a palliative care communication intervention.
      • Bernstein E.
      • Bernstein J.
      • Levenson S.
      Project ASSERT: An ED-based intervention to increase access to primary care, preventive services, and the substance abuse treatment system.
      The intervention's goal is to increase patient motivation to initiate ACP conversations with their primary outpatient clinicians. In our BNI intervention, ED clinicians ask patients if they have discussed their values and priorities for their health care with their primary outpatient clinician. Interventionists then follow a BNI framework to understand patients' perspectives and empower them to hold such conversations in the future. The intervention aims neither to change “in-the-moment” decision making nor to engage in actual ACP in the ED. Instead, it is structured to “talk about talking about it.” After each patient receives the intervention, the study PI contacts the primary outpatient clinician with a standardized e-mail or phone call. The PI informs the clinician about the patient's enrollment in the study and describes the study's aims. In addition, the study PI provides outpatient clinicians with information about continuing ACP conversations with patients and with a guide to serious illness conversation.
      • Bernacki R.
      • Hutchings M.
      • Vick J.
      • et al.
      Development of the serious illness care program: a randomised controlled trial of a palliative care communication intervention.
      To refine the BNI intervention, we conducted a series of mock clinical encounters between ED clinicians and members of the hospital's Patient Family Advisory Council (PFAC). The PFAC consists of patients, family members, and executive leaders who collaborate to improve hospital programs, policies, and patient care. PFAC members provide the patient's perspective and regularly work with clinicians on quality improvement, patient engagement, and research. After these mock encounters, we asked PFAC members to reflect on experiences they had in the ED and what it was like to receive care there. We then elicited their feedback about the intervention's acceptability from this perspective in cognitive interviews. We iteratively refined the intervention until we reached thematic saturation from the interviews and could no longer identify modifications to be made to the BNI intervention.

      Fidelity Study

      Once the intervention was refined, we conducted a pilot study in the ED with seriously ill older adults. Our institutional review board approved this study. We used convenience sampling to recruit patients in the ED from September to December 2017. We included English-speaking patients ≥65 years old with serious, life-limiting illness (metastatic cancer, oxygen-dependent chronic obstructive lung disease, chronic kidney disease on dialysis, New York Heart Association Stage 3 or 4 heart failure). We also included patients ≥18 years old if a treating ED clinician “would not be surprised if the patient died in the next 12 months.”
      • Ouchi K.
      • Jambaulikar G.
      • George N.R.
      • et al.
      The “surprise question” asked of emergency physicians may predict 12-month mortality among older emergency department patients.
      We excluded patients who had a medical order for life-sustaining treatment in the electronic health record, were determined by the treating ED clinicians to be inappropriate for this study (e.g., in acute physical or emotional distress), or could not provide informed consent.
      The PI (K. O.) trained two other interventionists (A. C. R. and L. A. F.), who were both physician assistants without prior experience in palliative care. The PI is an ED physician who has undergone extensive training in serious illness communication through participation in well-established and recognized courses.
      VitalTalk
      FAQ faculty development.
      Education in Palliative and End-of-Life Care
      EPEC-emergency medicine.
      Harvard Medical School Center for Palliative Care. Palliative Care Education and Practice (PCEP).
      He developed the training with members of the expert panel, drawing on their expertise in serious illness communication pedagogy. The Brief Negotiated Intervention–Active Referral to Treatment Institute's alcohol abuse intervention served as the model for clinician training.
      • Bernstein E.
      • Bernstein J.
      • Feldman J.
      • et al.
      An evidence based alcohol screening, brief intervention and referral to treatment (SBIRT) curriculum for emergency department (ED) providers improves skills and utilization.
      The training included three components: 1) didactic training on motivational interviewing; 2) serious illness communication training with professional actors that focuses on responding to emotion, eliciting what is most important, and respectfully introducing future care planning; and 3) bedside coaching until competency is demonstrated in more than five consecutive patient encounters. Clinicians underwent two hours of training, which did not include time spent receiving bedside coaching.
      We video-recorded all encounters. We developed a fidelity checklist based on those used to assess similar BNI interventions for alcohol use disorders in the ED.
      • D'Onofrio G.
      • Pantalon M.V.
      • Degutis L.C.
      • Fiellin D.A.
      • O'Connor P.G.
      Development and implementation of an emergency practitioner-performed brief intervention for hazardous and harmful drinkers in the emergency department.
      The checklist was adapted to fit the content of our ACP intervention by the study's PI, a palliative care physician (R. E. L.), a clinical psychologist with expertise in motivational interviewing (M. Y.), and a palliative care physician with expertise in clinician-patient communication research (J. A. T.). The fidelity checklist (Appendix 1) is organized into two parts: I) BNI steps; II) communication skills. Part I consists of six domains: Opening (e.g., a mention of the research on the importance of communicating care preferences), Rapport building (e.g., ask open-ended questions), Information & feedback (e.g., elicit current thoughts about discussing goals of care), Readiness (e.g., asking “How ready are you now?”), Summary (e.g., summarize patient's responses), and Action (e.g., elicit specific next step from patient's perspective). Each domain has two to three subdomains, which are scored dichotomously (yes/no). The clinician administering the intervention could receive up to 15 points for Part I.
      Part II consists of six domains: Language appropriateness, Reflective listening, Use of empathic statements, Assessment of mutual understanding, Listening for cues, and Redirects when needed. We scored each domain on a three-point scale: 0 if the skill was not demonstrated at all; 1 if the skill was demonstrated occasionally or demonstrated with fair quality; and 2 if the skill was demonstrated often and with high quality. If there were no opportunities for redirection, we scored this as 2. The clinician administering the intervention could receive up to 12 points for Part II and therefore could receive a maximum of 27 points for each interview. While there is no gold standard definition of high fidelity to a behavioral intervention, experts generally consider 70%–100% adherence to key components of the intervention to meet criteria for high fidelity.
      • Czajkowski S.M.
      • Powell L.H.
      • Adler N.
      • et al.
      From ideas to efficacy: the ORBIT model for developing behavioral treatments for chronic diseases.
      • Borrelli B.
      The assessment, monitoring, and enhancement of treatment fidelity in public health clinical trials.
      For this proof-of-concept study (Stage IIa of behavioral intervention development), we proactively determined that an average fidelity score of 19 points (70%) or higher would meet criteria for high fidelity to the intervention.
      Two reviewers with diverse clinical backgrounds (R. E. L. and M. Y.) watched video recordings of the interviews and completed fidelity checklists. They established operational definitions of what constituted “adherence” for each item before reviewing the recordings and reviewed six recordings together to establish consistency of coding. Each reviewer was responsible for 23 of 46 total recordings. The reviewers watched each recording at least twice, the first time to code Part I and the second time to code Part II. The reviewers could watch the recordings as many times as necessary to feel confident in their ratings. To assess interrater reliability (IRR), 15% of the recordings (n = 6) were assessed by both reviewers, which is consistent with published methods that double code 10%–20%.
      • Apodaca T.R.
      • Magill M.
      • Longabaugh R.
      • Jackson K.M.
      • Monti P.M.
      Effect of a significant other on client change talk in motivational interviewing.
      • Gaume J.
      • Bertholet N.
      • Faouzi M.
      • Gmel G.
      • Daeppen J.-B.
      Counselor motivational interviewing skills and young adult change talk articulation during brief motivational interventions.
      • Mitcheson L.
      • Bhavsar K.
      • McCambridge J.
      Randomized trial of training and supervision in motivational interviewing with adolescent drug treatment practitioners.
      • Moyers T.B.
      • Martin T.
      Therapist influence on client language during motivational interviewing sessions.
      • Moyers T.B.
      • Martin T.
      • Christopher P.J.
      • et al.
      Client language as a mediator of motivational interviewing efficacy: where is the evidence?.
      • Moyers T.B.
      • Manuel J.K.
      • Wilson P.G.
      • et al.
      A randomized trial investigating training in motivational interviewing for behavioral health providers.
      The IRR was the percent agreement between the reviewers on all scores. To calculate percent agreement, we added the number of times Reviewers 1 and 2 agreed on the same data item and divided that sum by the total number of data items.
      • Burns M.K.
      How to establish interrater reliability.
      The reviewers resolved discrepancies in ratings by consensus. If they could not reach consensus, a third reviewer (K. O.) scored the recording for the items in question.
      We calculated descriptive and summary statistics for the checklist as a whole and also for each of the two parts. Furthermore, the following Part I variables were recoded to represent a total score for each domain: Opening (three competencies), Information & Feedback (three competencies), Readiness (three competencies), Summary (two competencies), and Action (three competencies). For example, the “Readiness” domain included three separate competencies: 1) assess readiness; 2) reinforce positives; and 3) ask what would make you more ready. We added the scores for the three competencies to generate one total “Readiness” competency score.
      To assess feasibility, we calculated the percent of eligible patients who consented to participate in the study. We also measured the time spent in each encounter and the number of times that the interventionist was interrupted by another ED clinician or staff member. We used the Statistical Package for Social Sciences Version 24.0 (SPSS 24.0; IBM Corp., Armonk, NY) for all analyses.

      Results

      Sample Description

      We summarize the characteristics of the 46 study participants included in final analyses in Table 1. Participants were 51.1% female (n = 23) and predominantly white (82.2%). Ages ranged from 39 to 94 (M = 74.29, SD = 12.25). Metastatic cancer (51.1%) was the most common diagnosis, followed by <12-month predicted mortality (26.7%), New York Heart Association (NYHA) Class III/IV congestive heart failure (CHF, 8.9%), oxygen-dependent chronic obstructive pulmonary disease (COPD, 6.7%), and chronic kidney disease (CKD) on dialysis (6.7%). The Emergency Severity Index (ESI) is a five-level triage tool that categorizes patients into five groups, ranging from 1 (most urgent) to 5 (least urgent).
      Agency for Healthcare Research and Quality
      Emergency Severity Index (ESI) A Triage Tool for Emergency Department Care Implementation Handbook 2012 Edition.
      All enrolled patients had an ESI of either 2 (emergent, 52%) or 3 (urgent, 48%). The majority of patients (54%) were admitted to the hospital. A minority of patients (11%) were discharged home from the ED. The remainder of the enrolled patients (35%) were admitted to the ED observation unit. Of these patients, 69% were ultimately discharged, 25% were admitted, and one patient was transferred to another hospital.
      Table 1Baseline Characteristics of Study Participants
      Characteristics, N = 46 (%)
      Age, mean (SD)74.2 (12.3)
      Sex, n (%)
       Female24 (52.2)
      Race, n (%)
       White37 (80.4)
       Black5 (10.9)
       Other4 (8.7)
      Ethnicity, n (%)
       Hispanic3 (6.5)
      Life-limiting condition, n (%)
       Metastatic cancer23 (50.0)
       Chronic kidney disease (CKD) on dialysis3 (6.5)
       Oxygen-dependent chronic obstructive pulmonary disease (COPD)3 (6.5)
       New York Heart Association III/IV congestive heart failure (CHF)4 (8.7)
       <12-month predicted mortality
      ED clinician would not be “surprised if died in the next 12 months.”
      13 (28.3)
      Emergency Severity Index
      ESI is a five-level ED triage tool that categorizes patients into five groups, ranging from 1 (most urgent) to 5 (least urgent).42
      (ESI), n (%)
       2 (emergent)24 (52.1)
       3 (urgent)22 (47.8)
      Postintervention disposition, n (%)
       Home5 (10.9)
       Inpatient25 (54.3)
       ED observation16 (34.8)
      Home11 (68.8)
      Inpatient4 (25)
      Transfer1 (6.3)
      CKD = chronic kidney disease; COPD = chronic obstructive pulmonary disease; NYHA = New York Heart Association; CHF = congestive heart failure; ED = emergency department.
      a ED clinician would not be “surprised if died in the next 12 months.”
      b ESI is a five-level ED triage tool that categorizes patients into five groups, ranging from 1 (most urgent) to 5 (least urgent).
      Agency for Healthcare Research and Quality
      Emergency Severity Index (ESI) A Triage Tool for Emergency Department Care Implementation Handbook 2012 Edition.

      Interrater Reliability

      Reviewers double-coded six (15%) of the total recordings to assess IRR. To address the Cicchetti Paradox, which produces a low Kappa coefficient despite high interrater agreement, particularly in a small sample, we assessed IRR using percentages.
      • Feinstein A.R.
      • Cicchetti D.V.
      High agreement but low Kappa: I. the problems of two paradoxes.
      • Cicchetti D.V.
      • Feinstein A.R.
      High agreement but low Kappa.
      • Cicchetti D.V.
      When diagnostic agreement is high, but reliability is low: some paradoxes occurring in joint independent neuropsychology assessments.
      Overall, IRR was 90.6%. IRR for Part I (BNI steps) was 92.2%. IRR for Part II (communication skills) was 89%. The two primary reviewers reached consensus in all cases and did not require the third reviewer to intervene.

      Fidelity

      We summarize the results for intervention fidelity and feasibility in Table 2. Total fidelity ranged from 11 to 27 out of a total possible 27 points. The mean total score was 21.07 (SD = 3.68), or 78.04%. The total Part I (BNI steps) score ranged from 8 to 15 out of a total 15 points. The Part I mean score was 12.07 (SD = 2.07), or 80.47%. The Part II (communication skills) score ranged from 0 to 12 out of a total possible 12 points. The Part II mean score was 9.0 (SD = 2.51), or 75%. Of the 46 recordings analyzed, 34 (75.6%) received a score of at least 70% (19/27 or higher), thus meeting the prespecified fidelity threshold.
      Table 2Intervention Fidelity and Feasibility
      Fidelity % Scores, Mean (SD)
      Part I—BNI steps (max pts)
       Opening (three pts)3 (0)
       Rapport building (one pt)0.96 (.21)
       Information and feedback (three pts)2.36 (.71)
       Readiness (three pts)2.20 (.87)
       Summary (two pts)1.64 (.65)
       Action (three pts)1.95 (.96)
       Part I score (15 pts)12.07 (2.07)
      Part II—communication skills (two pts)
       Appropriate language1.45 (.50)
       Reflective listening1.82 (.39)
       Use of empathic language1.19 (.63)
       Assessing mutual understanding1.73 (.45)
       Listening for cues1.39 (.58)
       Redirects when needed1.73 (.54)
       Part II score (12 pts)9.00 (2.51)
      Overall score (27 pts)21.07 (3.68)
      Encounters with high fidelity
      Total score >70% (≥19/27).
      , n (%)
      34 (75.6)
      Feasibility
       Average time/encounter in minutes, median (interquartile range)10.5 (7.5–13.5)
       Interruptions, mean (SD)0.4 (0.7)
      a Total score >70% (≥19/27).

      Feasibility

      We screened 223 subjects for participation in the study. Of these, 84 met inclusion criteria and were approached for participation. Fifty subjects consented and were enrolled. The 34 subjects who did not consent were similar in demographics to those who enrolled. Most declined to participate owing to feeling weak or fatigued. Of those who enrolled, two subsequently withdrew consent. A third subject had a previously completed medical order for life-sustaining treatment on file in the EMR and was withdrawn from the study by the investigators. One subject was also enrolled but could not be included in our results because the video-recording equipment malfunctioned. Therefore, 46 (55%) of eligible patients completed the intervention and were included in study analyses. The median length of the BNI intervention was 10.5 minutes (interquartile range 7.5–13.5 minutes). The number of interruptions ranged from zero to three (M = 0.4, SD = 0.7).

      Associations With Intervention Fidelity

      Chi-square tests revealed no association between clinician type (physician assistant or physician) and intervention fidelity (yes/no), χ2 (1, n = 45) = 3.15, P = 0.08. Chi-square tests also revealed no association between coder background (palliative care physician/psychologist) and intervention fidelity (yes/no), χ2 (1, n = 39) = 0.01, P = 0.93.

      Discussion

      We demonstrated that trained ED clinicians can conduct the BNI intervention to increase ACP conversations with high fidelity. Most encounters met criteria for high fidelity to the intervention. Clinicians adhered to both the BNI steps and the serious illness communication components of the intervention. The majority of eligible patients consented to participate and interventions were appropriately brief in duration.
      Our study showed that it is possible to engage seriously ill patients in a discussion on the importance of ACP in the ED. We were able to approach all eligible patients and over half (55%) consented to the study and underwent the intervention. Enrolled patients were quite ill—all had an ESI of either 2 or 3 and disposition for a majority (54%) was hospital admission. The high enrollment rate suggests that seriously ill patients are willing and able to participate in a BNI ACP intervention, despite numerous barriers.
      • Ekwall A.
      Acuity and anxiety from the patient's perspective in the emergency department.
      • Byrne G.
      • Heyman R.
      Patient anxiety in the accident and emergency department.
      • Rising K.L.
      • Hudgins A.
      • Reigle M.
      • Hollander J.E.
      • Carr B.G.
      “I'm just a patient”: fear and uncertainty as drivers of emergency department use in patients with chronic disease.
      We are unable to evaluate how many full ACP conversations actually occurred for the 55% of patients who participated in the study. However, increasing the number of seriously ill patients seen in the ED who are primed, and whose clinicians are primed to conduct an ACP conversation, has potential to reduce the number of these “at-risk” patients who visit the ED without ACP in the future.
      An ED visit can serve as a “teachable moment” for seriously ill patients. Teachable moments describe health events that motivate individuals to adopt risk-reducing behaviors.
      • McBride C.M.
      • Emmons K.M.
      • Lipkus I.M.
      Understanding the potential of teachable moments: the case of smoking cessation.
      Investigators have used this concept to encourage a variety of behavior changes, such as tobacco cessation and pediatric screen time reduction.
      • McBride C.M.
      • Emmons K.M.
      • Lipkus I.M.
      Understanding the potential of teachable moments: the case of smoking cessation.
      • Bluethmann S.M.
      • Basen-Engquist K.
      • Vernon S.W.
      • et al.
      Grasping the ‘teachable moment’: time since diagnosis, symptom burden and health behaviors in breast, colorectal and prostate cancer survivors.
      • Erkoboni D.
      • Radesky J.
      The elephant in the examination room: addressing parent and child mobile device use as a teachable moment.
      In order for an event to be a teachable moment, it should 1) increase perceptions of personal risk and outcome expectancies; 2) prompt a strong affective or emotional response, and 3) redefine self-concept or social role.
      • McBride C.M.
      • Emmons K.M.
      • Lipkus I.M.
      Understanding the potential of teachable moments: the case of smoking cessation.
      A visit to the ED by a seriously ill patient meets all three criteria and can be used to motivate patients to avoid similar situations in the future should they not be compatible with their goals and values. Our BNI intervention leverages the ED teachable moment and also allows for the involvement of an outpatient clinician with whom a patient has greater continuity and trust.
      • Lakin J.R.
      • Block S.D.
      • Billings J.A.
      • et al.
      Improving communication about serious illness in primary care.
      ED clinicians worry about the time investment ACP requires and their lack of adequate skills to carry out these interventions.
      • Smith A.K.
      • Fisher J.
      • Schonberg M.A.
      • et al.
      Am I doing the right thing? Provider perspectives on improving palliative care in the emergency department.
      • Mchugh M.
      • Dyke K. Van
      • Mcclelland M.
      • Moss D.
      Improving Patient Flow and Reducing Emergency Department Crowding: A Guide for Hospitals.
      In our study, the median intervention lasted 10.5 minutes, which is consistent with other ED BNI interventions.
      • D'Onofrio G.
      • Fiellin D.A.
      • Pantalon M.V.
      • et al.
      A brief intervention reduces hazardous and harmful drinking in emergency department patients.
      • D'Onofrio G.
      • Pantalon M.V.
      • Degutis L.C.
      • Fiellin D.A.
      • O'Connor P.G.
      Development and implementation of an emergency practitioner-performed brief intervention for hazardous and harmful drinkers in the emergency department.
      • D'Onofrio G.
      • O'Connor P.G.
      • Pantalon M.V.
      • et al.
      Emergency department-initiated buprenorphine/naloxone treatment for opioid dependence: a randomized clinical trial.
      After a brief, systematic training, participating clinicians received high scores on their adherence to the steps of the intervention and on their communication with patients. Skilled communication is critical in encounters with seriously ill patients that deal with hopes, worries, goals, and values.
      • Steinhauser K.E.
      • Christakis N.A.
      • Clipp E.C.
      • McNeilly M.
      • McIntyre L.
      • Tulsky J.A.
      Factors considered important at the end of life by patients, family, physicians, and other care providers.
      • Back A.L.
      • Arnold R.M.
      • Tulsky J.A.
      • Baile W.F.
      • Fryer-Edwards K.A.
      Teaching communication skills to medical oncology fellows.
      • Tulsky J.A.
      • Arnold R.M.
      • Alexander S.C.
      • et al.
      Enhancing communication between oncologists and patients with a computer-based training program.
      • Tulsky J.A.
      • Beach M.C.
      • Butow P.N.
      • et al.
      A research agenda for communication between health care professionals and patients living with serious illness.
      Inexpert communication may leave patients confused or, worse, with increased distress.
      • Wenrich M.D.
      • Curtis J.R.
      • Shannon S.E.
      • et al.
      Communicating with dying patients within the spectrum of medical care from terminal diagnosis to death.
      • Tulsky J.A.
      • Chesney M.A.
      • Lo B.
      How do medical residents discuss resuscitation with patients?.
      • Hanson L.C.
      • Danis M.
      • Garrett J.
      What is wrong with end-of-life care? Opinions of bereaved family members.
      • Tulsky J.A.
      • Fischer G.S.
      • Rose M.R.
      • Arnold R.M.
      Opening the black box: how do physicians communicate about advance directives?.
      Serious illness communication skills can be taught,
      • Back A.L.
      • Arnold R.M.
      • Tulsky J.A.
      • Baile W.F.
      • Fryer-Edwards K.A.
      Teaching communication skills to medical oncology fellows.
      • Lakin J.R.
      • Koritsanszky L.A.
      • Cunningham R.
      • et al.
      A systematic intervention to improve serious illness communication in primary care.
      • Tulsky J.A.
      • Arnold R.M.
      • Alexander S.C.
      • et al.
      Enhancing communication between oncologists and patients with a computer-based training program.
      • Jacobsen J.
      • Brenner K.
      • Greer J.A.
      • et al.
      When a patient is reluctant to talk about it: a dual framework to focus on living well and tolerate the possibility of dying.
      • Schell J.O.
      • Green J.A.
      • Tulsky J.A.
      • Arnold R.M.
      Communication skills training for dialysis decision-making and end-of-life care in nephrology.
      and these pedagogic methods should be incorporated into BNI interventions for ACP.
      Beyond clinician time and training, traditional payment models can disincentivize emergency clinicians from engaging in ACP. Changes implemented in 2016 by the Center for Medicare and Medicaid Services, however, allow clinicians to submit charges for time spent facilitating ACP with their patients across settings, including in the ED. Accountable care organizations, value-based payment, and other risk-sharing models also encourage administrators to adopt population health management strategies at the health system level.
      • Colla C.H.
      • Lewis V.A.
      • Kao L.-S.
      • et al.
      Association between Medicare accountable care organization implementation and spending among clinically vulnerable beneficiaries.
      • Nyweide D.J.
      • Lee W.
      • Cuerdon T.T.
      • et al.
      Association of pioneer accountable care organizations vs traditional Medicare fee for service with spending, utilization, and patient experience.
      Increasing both access to palliative care and completion of ACP has potential to add significant value.
      • Lakin J.R.
      • Koritsanszky L.A.
      • Cunningham R.
      • et al.
      A systematic intervention to improve serious illness communication in primary care.
      • Bond W.F.
      • Kim M.
      • Franciskovich C.M.
      • et al.
      Advance care planning in an accountable care organization is associated with increased advanced directive documentation and decreased costs.
      • Ahluwalia S.C.
      • Harris B.J.
      • Lewis V.A.
      • Colla C.H.
      End-of-life care planning in accountable care organizations: associations with organizational characteristics and capabilities.
      • Smith G.
      • Bernacki R.
      • Block S.D.
      The role of palliative care in population management and accountable care organizations.
      If effective and integrated system-wide, the BNI approach to increase ACP could provide cost and quality incentives for the health care system over time.
      This study has several limitations and its results must be interpreted in the context of its design. First, the study was designed to assess intervention fidelity and feasibility in a single ED setting. We did not assess the efficacy of the intervention on increasing ACP conversations between patients and their primary outpatient clinicians, a study that is currently ongoing. Moreover, we do not assess the intervention's potential harms, although neither BNI nor ACP interventions have been previously shown to produce negative effects on patients.
      • D'Onofrio G.
      • Fiellin D.A.
      • Pantalon M.V.
      • et al.
      A brief intervention reduces hazardous and harmful drinking in emergency department patients.
      • Lakin J.R.
      • Koritsanszky L.A.
      • Cunningham R.
      • et al.
      A systematic intervention to improve serious illness communication in primary care.
      • D'Onofrio G.
      • Degutis L.C.
      Integrating Project ASSERT: a screening, intervention, and referral to treatment program for unhealthy alcohol and drug use into an urban emergency department.
      Second, the study involved only three interventionists, one of whom designed the intervention. Fidelity may change as the number and heterogeneity of clinicians performing the intervention increases. Our goal was simply to see whether it was feasible to deliver the intervention with high fidelity. However, two of the three interventionists were physician assistants without prior experience or training in palliative care or serious illness communication. This suggests that it may be possible to disseminate the intervention with a wider variety of interventionists. Third, the study was performed in a single ED at a large, urban tertiary care academic medical center with a high proportion of patients with cancer and a relatively homogenous, white population. Selection of participants by convenience sampling may have also biased our results. Future prospective testing will elucidate whether the intervention is both effective and scalable to different settings and patient populations.

      Conclusions

      A BNI intervention may be able to harness the ED visit by seriously ill patients as a teachable moment for ACP. Although barriers exist to primary palliative care interventions in the ED, this study demonstrated that ED clinicians can deliver a BNI intervention designed to increase ACP conversations between seriously ill patients and their outpatient clinicians with high fidelity. Trained ED clinicians followed the steps of the intervention and demonstrated appropriate communication skills. Future research will study the efficacy of our ED BNI intervention on patient and clinician satisfaction, completion of ACP conversations and documentation, and end-of-life outcomes in a seriously ill patient population.

      Disclosures and Acknowledgments

      The authors wish to acknowledge Ms. Sarah Pajka, BS, for her contributions to this study.
      This work was supported by the Grants for Early Medical and Surgical Subspecialists' Transition to Aging Research (GEMSSTAR) award (R03AG056449), from the National Institute on Aging. A supplemental grant from the Emergency Medicine Foundation, and the Society of Academic Emergency Medicine also supported this work. All authors report no conflicts of interest.

      Appendix A. Supplementary data

      References

        • Smith A.K.
        • McCarthy E.
        • Weber E.
        • et al.
        Half of older Americans seen in emergency department in last month of life; most admitted to hospital, and many die there.
        Health Aff. 2012; 31: 1277-1285
        • Steinhauser K.E.
        • Christakis N.A.
        • Clipp E.C.
        • McNeilly M.
        • McIntyre L.
        • Tulsky J.A.
        Factors considered important at the end of life by patients, family, physicians, and other care providers.
        JAMA. 2000; 284: 2476-2482
        • Oulton J.
        • Rhodes S.M.
        • Howe C.
        • Fain M.J.
        • Mohler M.J.
        Advance directives for older adults in the emergency department: a systematic review.
        J Palliat Med. 2015; 18: 500-505
        • Lakin J.R.
        • Block S.D.
        • Billings J.A.
        • et al.
        Improving communication about serious illness in primary care.
        JAMA Intern Med. 2016; 176: 1380
        • Bernacki R.E.
        • Block S.D.
        Communication about serious illness care goals: a review and synthesis of best practices.
        JAMA Intern Med. 2014; 174: 1994-2003
        • Shen M.J.
        • Prigerson H.G.
        • Paulk E.
        • et al.
        Impact of end-of-life discussions on the reduction of Latino/non-Latino disparities in do-not-resuscitate order completion.
        Cancer. 2016; 122: 1749-1756
        • Khandelwal N.
        • Kross E.K.
        • Engelberg R.A.
        • et al.
        Estimating the effect of palliative care interventions and advance care planning on ICU utilization.
        Crit Care Med. 2015; 43: 1102-1111
        • Stone S.C.
        • Mohanty S.
        • Grudzen C.R.
        • et al.
        Emergency medicine physicians' perspectives of providing palliative care in an emergency department.
        J Palliat Med. 2011; 14: 1333-1338
        • Wright R.J.
        • Lowton K.
        • Robert G.
        • Grudzen C.R.
        • Grocott P.
        Emergency department staff priorities for improving palliative care provision for older people: a qualitative study.
        Palliat Med. 2018; 32: 417-425
        • Smith A.K.
        • Fisher J.
        • Schonberg M.A.
        • et al.
        Am I doing the right thing? Provider perspectives on improving palliative care in the emergency department.
        Ann Emerg Med. 2009; 54 (93.e1): 86-93
        • Rollnick S.
        • Miller W.R.
        What is motivational interviewing?.
        Behav Cogn Psychother. 1995; 23: 325-334
        • Hibbard J.H.
        • Greene J.
        What the evidence shows about patient activation: better health outcomes and care experiences; fewer data on costs.
        Health Aff. 2013; 32: 207-214
        • D'Onofrio G.
        • Fiellin D.A.
        • Pantalon M.V.
        • et al.
        A brief intervention reduces hazardous and harmful drinking in emergency department patients.
        Ann Emerg Med. 2012; 60: 181-192
        • Bernstein E.
        • Edwards E.
        • Dorfman D.
        • et al.
        Screening and brief intervention to reduce Marijuana use among youth and young adults in a pediatric emergency department.
        Acad Emerg Med. 2009; 16: 1174-1185
        • Bernstein J.
        • Bernstein E.
        • Tassiopoulos K.
        • et al.
        Brief motivational intervention at a clinic visit reduces cocaine and heroin use.
        Drug Alcohol Depend. 2005; 77: 49-59
        • Bruguera P.
        • Barrio P.
        • Oliveras C.
        • et al.
        Effectiveness of a specialized brief intervention for at risk drinkers in an emergency department: short term results of a randomized controlled trial.
        Acad Emerg Med. 2018; 25: 517-525
        • Barata I.A.
        • Shandro J.R.
        • Montgomery M.
        • et al.
        Effectiveness of SBIRT for alcohol use disorders in the emergency department: a systematic review.
        West J Emerg Med. 2017; 18: 1143-1152
        • Fried T.R.
        • Leung S.L.
        • Blakley L.A.
        • Martino S.
        Development and pilot testing of a motivational interview for engagement in advance care planning.
        J Palliat Med. 2018; 21: 897-898
        • Beebe J.
        Rapid qualitative inquiry: A field guide to team-based assessment.
        Rowman & Littlefield, London2014
        • Bernstein E.
        • Bernstein J.
        • Feldman J.
        • et al.
        An evidence based alcohol screening, brief intervention and referral to treatment (SBIRT) curriculum for emergency department (ED) providers improves skills and utilization.
        Subst Abus. 2007; 28: 79-92
        • Back A.L.
        • Arnold R.M.
        • Tulsky J.A.
        • Baile W.F.
        • Fryer-Edwards K.A.
        Teaching communication skills to medical oncology fellows.
        J Clin Oncol. 2003; 21: 2433-2436
        • Pollak K.I.
        • Back A.L.
        • Tulsky J.A.
        Disseminating effective clinician communication techniques: engaging clinicians to want to learn how to engage patients.
        Patient Educ Couns. 2017; 100: 1951-1954
        • Back A.L.
        • Arnold R.M.
        • Baile W.F.
        • et al.
        Faculty development to change the paradigm of communication skills teaching in oncology.
        J Clin Oncol. 2009; 27: 1137-1141
        • Cooper Z.
        • Koritsanszky L.A.
        • Cauley C.E.
        • et al.
        Recommendations for best communication practices to facilitate goal-concordant care for seriously ill older patients with emergency surgical conditions.
        Ann Surg. 2016; 263: 1-6
        • Lakin J.R.
        • Koritsanszky L.A.
        • Cunningham R.
        • et al.
        A systematic intervention to improve serious illness communication in primary care.
        Health Aff. 2017; 36: 1258-1264
        • Bernacki R.
        • Hutchings M.
        • Vick J.
        • et al.
        Development of the serious illness care program: a randomised controlled trial of a palliative care communication intervention.
        BMJ Open. 2015; 5: e009032
        • Bernstein E.
        • Bernstein J.
        • Levenson S.
        Project ASSERT: An ED-based intervention to increase access to primary care, preventive services, and the substance abuse treatment system.
        (Available from) (Accessed August 28, 2018)
        • Ouchi K.
        • Jambaulikar G.
        • George N.R.
        • et al.
        The “surprise question” asked of emergency physicians may predict 12-month mortality among older emergency department patients.
        J Palliat Med. 2018; 21: 236-240
        • VitalTalk
        FAQ faculty development.
        (Available from) (Accessed August 27, 2018)
        • Education in Palliative and End-of-Life Care
        EPEC-emergency medicine.
        (Available from) (Accessed August 27, 2018)
      1. Harvard Medical School Center for Palliative Care. Palliative Care Education and Practice (PCEP).
        (Available from) (Accessed August 27, 2018)
        • D'Onofrio G.
        • Pantalon M.V.
        • Degutis L.C.
        • Fiellin D.A.
        • O'Connor P.G.
        Development and implementation of an emergency practitioner-performed brief intervention for hazardous and harmful drinkers in the emergency department.
        Acad Emerg Med. 2005; 12: 249-256
        • Czajkowski S.M.
        • Powell L.H.
        • Adler N.
        • et al.
        From ideas to efficacy: the ORBIT model for developing behavioral treatments for chronic diseases.
        Heal Psychol. 2015; 34: 971-982
        • Borrelli B.
        The assessment, monitoring, and enhancement of treatment fidelity in public health clinical trials.
        J Public Health Dent. 2011; 71: S52-S63
        • Apodaca T.R.
        • Magill M.
        • Longabaugh R.
        • Jackson K.M.
        • Monti P.M.
        Effect of a significant other on client change talk in motivational interviewing.
        J Consult Clin Psychol. 2013; 81: 35-46
        • Gaume J.
        • Bertholet N.
        • Faouzi M.
        • Gmel G.
        • Daeppen J.-B.
        Counselor motivational interviewing skills and young adult change talk articulation during brief motivational interventions.
        J Subst Abuse Treat. 2010; 39: 272-281
        • Mitcheson L.
        • Bhavsar K.
        • McCambridge J.
        Randomized trial of training and supervision in motivational interviewing with adolescent drug treatment practitioners.
        J Subst Abuse Treat. 2009; 37: 73-78
        • Moyers T.B.
        • Martin T.
        Therapist influence on client language during motivational interviewing sessions.
        J Subst Abuse Treat. 2006; 30: 245-251
        • Moyers T.B.
        • Martin T.
        • Christopher P.J.
        • et al.
        Client language as a mediator of motivational interviewing efficacy: where is the evidence?.
        Alcohol Clin Exp Res. 2007; 31: 40s-47s
        • Moyers T.B.
        • Manuel J.K.
        • Wilson P.G.
        • et al.
        A randomized trial investigating training in motivational interviewing for behavioral health providers.
        Behav Cogn Psychother. 2008; 36: 149-162
        • Burns M.K.
        How to establish interrater reliability.
        Nursing (Lond). 2014; 44: 56-58
        • Agency for Healthcare Research and Quality
        Emergency Severity Index (ESI) A Triage Tool for Emergency Department Care Implementation Handbook 2012 Edition.
        2012 (Available from) (Accessed August 27, 2018)
        • Feinstein A.R.
        • Cicchetti D.V.
        High agreement but low Kappa: I. the problems of two paradoxes.
        J Clin Epidemiol. 1990; 43: 543-549
        • Cicchetti D.V.
        • Feinstein A.R.
        High agreement but low Kappa.
        J Clin Epidemiol. 1990; 43: 551-585
        • Cicchetti D.V.
        When diagnostic agreement is high, but reliability is low: some paradoxes occurring in joint independent neuropsychology assessments.
        J Clin Exp Neuropsychol. 1988; 10: 605-622
        • Ekwall A.
        Acuity and anxiety from the patient's perspective in the emergency department.
        J Emerg Nurs. 2013; 39: 534-538
        • Byrne G.
        • Heyman R.
        Patient anxiety in the accident and emergency department.
        J Clin Nurs. 1997; 6: 289-295
        • Rising K.L.
        • Hudgins A.
        • Reigle M.
        • Hollander J.E.
        • Carr B.G.
        “I'm just a patient”: fear and uncertainty as drivers of emergency department use in patients with chronic disease.
        Ann Emerg Med. 2016; 68: 536-543
        • McBride C.M.
        • Emmons K.M.
        • Lipkus I.M.
        Understanding the potential of teachable moments: the case of smoking cessation.
        Health Educ Res. 2003; 18: 156-170
        • Bluethmann S.M.
        • Basen-Engquist K.
        • Vernon S.W.
        • et al.
        Grasping the ‘teachable moment’: time since diagnosis, symptom burden and health behaviors in breast, colorectal and prostate cancer survivors.
        Psychooncology. 2015; 24: 1250-1257
        • Erkoboni D.
        • Radesky J.
        The elephant in the examination room: addressing parent and child mobile device use as a teachable moment.
        J Pediatr. 2018; 198: 5-6
        • Mchugh M.
        • Dyke K. Van
        • Mcclelland M.
        • Moss D.
        Improving Patient Flow and Reducing Emergency Department Crowding: A Guide for Hospitals.
        2011 (Available from) (Accessed June 26, 2018)
        • D'Onofrio G.
        • O'Connor P.G.
        • Pantalon M.V.
        • et al.
        Emergency department-initiated buprenorphine/naloxone treatment for opioid dependence: a randomized clinical trial.
        JAMA. 2015; 313: 1636-1644
        • Tulsky J.A.
        • Arnold R.M.
        • Alexander S.C.
        • et al.
        Enhancing communication between oncologists and patients with a computer-based training program.
        Ann Intern Med. 2011; 155: 593-601
        • Tulsky J.A.
        • Beach M.C.
        • Butow P.N.
        • et al.
        A research agenda for communication between health care professionals and patients living with serious illness.
        JAMA Intern Med. 2017; 177: 1361-1366
        • Wenrich M.D.
        • Curtis J.R.
        • Shannon S.E.
        • et al.
        Communicating with dying patients within the spectrum of medical care from terminal diagnosis to death.
        Arch Intern Med. 2001; 161: 868-874
        • Tulsky J.A.
        • Chesney M.A.
        • Lo B.
        How do medical residents discuss resuscitation with patients?.
        J Gen Intern Med. 1995; 10: 436-442
        • Hanson L.C.
        • Danis M.
        • Garrett J.
        What is wrong with end-of-life care? Opinions of bereaved family members.
        J Am Geriatr Soc. 1997; 45: 1339-1344
        • Tulsky J.A.
        • Fischer G.S.
        • Rose M.R.
        • Arnold R.M.
        Opening the black box: how do physicians communicate about advance directives?.
        Ann Intern Med. 1998; 129: 441
        • Jacobsen J.
        • Brenner K.
        • Greer J.A.
        • et al.
        When a patient is reluctant to talk about it: a dual framework to focus on living well and tolerate the possibility of dying.
        J Palliat Med. 2018; 21: 322-327
        • Schell J.O.
        • Green J.A.
        • Tulsky J.A.
        • Arnold R.M.
        Communication skills training for dialysis decision-making and end-of-life care in nephrology.
        Clin J Am Soc Nephrol. 2013; 8: 675-680
        • American College of Emergency Physicians
        Advance Care Planning FAQ.
        2017 (Available from) (Accessed August 27, 2018)
        • Center for Medicare and Medicaid Services
        Advance Care Planning.
        2018 (Available from) (Accessed August 27, 2018)
        • Colla C.H.
        • Lewis V.A.
        • Kao L.-S.
        • et al.
        Association between Medicare accountable care organization implementation and spending among clinically vulnerable beneficiaries.
        JAMA Intern Med. 2016; 176: 1167-1175
        • Nyweide D.J.
        • Lee W.
        • Cuerdon T.T.
        • et al.
        Association of pioneer accountable care organizations vs traditional Medicare fee for service with spending, utilization, and patient experience.
        JAMA. 2015; 313: 2152-2161
        • Bond W.F.
        • Kim M.
        • Franciskovich C.M.
        • et al.
        Advance care planning in an accountable care organization is associated with increased advanced directive documentation and decreased costs.
        J Palliat Med. 2018; 21: 489-502
        • Ahluwalia S.C.
        • Harris B.J.
        • Lewis V.A.
        • Colla C.H.
        End-of-life care planning in accountable care organizations: associations with organizational characteristics and capabilities.
        Health Serv Res. 2018; 53: 1662-1681
        • Smith G.
        • Bernacki R.
        • Block S.D.
        The role of palliative care in population management and accountable care organizations.
        J Palliat Med. 2015; 18: 486-494
        • D'Onofrio G.
        • Degutis L.C.
        Integrating Project ASSERT: a screening, intervention, and referral to treatment program for unhealthy alcohol and drug use into an urban emergency department.
        Acad Emerg Med. 2010; 17: 903-911