Advertisement

Usage Patterns of a Web-Based Palliative Care Content Platform (PalliCOVID) During the COVID-19 Pandemic

      Abstract

      Context

      The COVID-19 pandemic has highlighted the essential role of palliative care to support the delivery of compassionate, goal-concordant patient care. We created the Web-based application, PalliCOVID (https://pallicovid.app/), in April 2020 to provide all clinicians with convenient access to palliative care resources and support. PalliCOVID features evidence-based clinical guidelines, educational content, and institutional protocols related to palliative care for COVID-19 patients. It is a publicly available resource accessible from any mobile device or desktop computer that provides clinicians with access to palliative care guidance across a variety of care settings, including the emergency department, hospital ward, intensive care unit, and primary care practice.

      Objective

      The primary objective of this study was to evaluate usage patterns of PalliCOVID to understand user behavior in relation to this palliative care content platform during the period of the local peak of COVID-19 infection in Massachusetts.

      Methods

      We retrospectively analyzed deidentified usage data collected by Google Analytics from the first day of PalliCOVID's launch on April 7, 2020, until May 1, 2020, the time period that encompassed the local peak of the COVID-19 surge in Massachusetts. User access data were collected and summarized by using Google Analytics software that had been integrated into the PalliCOVID Web application.

      Results

      A total of 2042 users accessed PalliCOVID and viewed 4637 pages from April 7 to May 1, 2020. Users spent an average of 2 minutes and 6 seconds per session. Eighty-one percent of users were first-time visitors, while the remaining 19% were return visitors. Most users accessed PalliCOVID from the United States (87%), with a large proportion of users coming from Boston and the surrounding cities (32% of overall users).

      Conclusions

      PalliCOVID is one example of a scalable digital health solution that can bring palliative care resources to frontline clinicians. Analysis of PalliCOVID usage patterns has the potential to inform the improvement of the platform to better meet the needs of its user base and guide future dissemination strategies. The quantitative data presented here, although informative about user behavior, should be supplemented with future qualitative research to further define the impact of this tool and extend our ability to deliver clinical care that is compassionate, rational, and well-aligned with patients’ values and goals.

      Key Words

      Introduction

      Background

      The novel coronavirus (COVID-19) pandemic has strained the health-care system beyond its usual capacity to accommodate such a large influx of seriously ill patients.
      • Curtis J.R.
      • Kross E.K.
      • Stapleton R.D.
      The importance of addressing advance care planning and decisions about do-not-resuscitate orders during novel coronavirus 2019 (COVID-19).
      ,
      • Fausto J.
      • Hirano L.
      • Lam D.
      • et al.
      Creating a palliative care inpatient response plan for COVID19 - the UW medicine experience.
      This has created the need to bring palliative care medicine practices to the frontline across specialties.
      • Heyland D.K.
      What matters most in end-of-life care: perceptions of seriously ill patients and their family members.
      • Goldstein J.
      • Rothfeld M.
      • Weiser B.
      Patient has virus and serious cancer. Should doctors withhold ventilator? The New York times.
      • Emanuel E.J.
      • Persad G.
      • Upshur R.
      • et al.
      Fair allocation of scarce medical resources in the time of Covid-19.
      • Arya A.
      • Buchman S.
      • Gagnon B.
      • Downar J.
      Pandemic palliative care: beyond ventilators and saving lives.
      Many clinicians who have not been specially trained in palliative care do not feel adequately prepared to facilitate difficult conversations with patients or their family members about goals of care.
      • Stone S.C.
      • Mohanty S.
      • Grudzen C.R.
      • et al.
      Emergency medicine physicians' perspectives of providing palliative care in an emergency department.
      Providing these clinicians with convenient access to focused education, clinical reference materials, and specialist palliative care support may improve end-of-life care for patients.
      • Curtis J.R.
      • Kross E.K.
      • Stapleton R.D.
      The importance of addressing advance care planning and decisions about do-not-resuscitate orders during novel coronavirus 2019 (COVID-19).
      As the COVID-19 surge approached Boston in April 2020, our interdisciplinary team of emergency physicians, palliative care specialists, and digital health innovators at Partners HealthCare recognized the need to rapidly increase the capacity to deliver primary palliative care at the frontline. Partners HealthCare is a large academic integrated health-care system that was founded by Massachusetts General Hospital and Brigham and Women's Hospital in 1994 and now consists of a core network of 11 hospitals in the New England area. Our goal was to create a centralized resource of information that would support Partners health-care staff to provide goal-concordant care and expert symptom management to patients who were ill with suspected COVID-19 infection.
      Given the challenges of distributing and maintaining the quality of up-to-date information in such a fast-changing environment, we sought out a digital solution. We aimed to develop an online, centralized compendium of clinical reference materials, focused educational content, and institutional protocols intended to seamlessly integrate palliative care across a variety of care settings, including the emergency department, hospital ward, intensive care unit, and primary care practice with respect to Partners-specific resources and established protocols.

      Methods

      Development of the Platform

      We built a progressive Web application (PWA) called “PalliCOVID” for the purpose of sharing digital content related to the delivery of palliative care during the COVID-19 pandemic. PWAs have previously been noted to have significant untapped potential in the field of health care to improve work efficiency and quality of care.
      • Rêgo F.
      • Portela F.
      • Santos M.F.
      Towards PWA in healthcare.
      The main advantages of PWAs are that they provide the user with a similar experience to using a native application but are accessed via a Web browser, do not require download from an app store, can be shared via a URL link, and can be found via a Web search engine. Users around the world can access PalliCOVID via Web browsers on any mobile or desktop device, thus allowing for wide dissemination of the digital content posted by the authors. At the same time, the authors are able to centrally manage the content and make instantaneous updates as needed, which is an important mechanism for quality control considering the rapidly changing information environment of the pandemic.
      The clinical guidelines and content featured on PalliCOVID were sourced, produced, and reviewed by palliative care specialists at Brigham and Women's Hospital and Dana-Farber Cancer Institute and emergency physicians from Brigham and Women's Hospital. This interdisciplinary team of nurses, physicians, and physician assistants met weekly to discuss updates to the content. We prioritized the publication of guidelines that were succinct, specific, and commensurate with the best available scientific evidence about the COVID-19 disease process and end-of-life care. We established several key criteria for content posted on PalliCOVID:
      • Accurate: Content was reviewed by palliative care experts to reflect the best available scientific evidence.
      • Practical: Recommendations were designed to be useful and implementable by nonpalliative care clinicians in a variety of care settings.
      • Accessible: Content was presented in a format that was optimized for viewing on both mobile devices and desktop computer screens.
      • Applicable: Content was specific to the care of patients with confirmed or suspected COVID-19 infection and took into account the need to limit face-to-face interactions because of enhanced infection-control measures and restricted visitor policies.
      We have included examples of PalliCOVID resources that respectively address the three major domains of palliative care expertise with regard to symptom management, goals of care discussions, and family support:
      • Arya A.
      • Buchman S.
      • Gagnon B.
      • Downar J.
      Pandemic palliative care: beyond ventilators and saving lives.
      ,
      • Hawley P.
      Barriers to access to palliative care.
      opioid dosing recommendations for the treatment of dyspnea and pain at the end of life (Fig. 1), a conversation guide for rapid code status determination in the peri-intubation setting (Fig. 2), and a tip sheet for responding to difficult questions from family members of dying patients (Fig. 3). Although certain features of PalliCOVID are restricted to users within the Partners HealthCare System, such as one-click access to the hospital paging system, we have made the majority of the content on PalliCOVID open to public access.
      Figure thumbnail gr1
      Fig. 1Opioids for treatment of dyspnea and pain pocket card.
      Figure thumbnail gr2
      Fig. 2Rapid code status determination conversation guide for use in peri-intubation situations in the emergency department.
      Figure thumbnail gr3
      Fig. 3Common questions from family members of imminently dying patients.

      Study Design

      The primary objective of this study was to collect and analyze usage data from PalliCOVID as a way to better understand user behavior and gain insights about the population of users accessing this palliative care content platform. The secondary objective was to use the insights derived from user engagement behavior to formulate ideas about future improvements to the platform and future dissemination strategies.
      We retrospectively analyzed deidentified usage data that were collected by Google Analytics from the first day of PalliCOVID's launch on April 7, 2020, through May 1, 2020. Google Analytics is a free tool that provides quantitative data on website usage and has previously been used in health research for process evaluation and quality improvement.
      • Costantini M.
      • Sleeman K.E.
      • Peruselli C.
      • Higginson I.J.
      Response and role of palliative care during the COVID-19 pandemic: a national telephone survey of hospices in Italy.
      This project was undertaken as a Quality Improvement Initiative at Brigham and Women's Hospital and as such was not formally supervised by the institutional review board per their policies. We followed the SQUIRE guidelines for quality improvement reporting.
      • Ogrinc G.
      • Davies L.
      • Goodman D.
      • Batalden P.
      • Davidoff F.
      • Stevens D.
      SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process: Table 1.

      Evaluating Usage

      We evaluated usage data using Google Analytics (Google, LLC, Mountain View, CA), which was installed on PalliCOVID and used to track user data from April 7, 2020, through May 1, 2020.
      • Song M.J.
      • Ward J.
      • Choi F.
      • et al.
      A process evaluation of a web-based mental health portal (WalkAlong) using Google Analytics.
      This allowed collection of data related to user behavior emanating from a user's interaction with the application. The data could come from avenues such as the URLs of the pages the user viewed, the location of the user, and the type of device being used to access the application. Google Analytics also collected information about the nature of the visit such as the content viewed and length of the session. Google Analytics did not collect any personally identifiable information and presented all collected data in aggregate form, thus mitigating the ethical concerns that could arise from user behavior research.
      • Clark D.J.
      • Nicholas D.
      • Jamali H.R.
      Evaluating information seeking and use in the changing virtual world: the emerging role of Google Analytics.
      ,
      • Kirk M.
      • Morgan R.
      • Tonkin E.
      • McDonald K.
      • Skirton H.
      An objective approach to evaluating an internet-delivered genetics education resource developed for nurses: using Google AnalyticsTM to monitor global visitor engagement.
      We evaluated user engagement by examining “sessions”. A session was defined as a series of interactions by a user that took place within a predetermined time frame (30 minutes), which represented the period of time that the user was actively engaged with the application. The number of returning users referred to the number of sessions visited by the same client ID. A relatively high proportion of returning users has been noted in Web-based mental health intervention research to be a marker of user engagement.
      • Costantini M.
      • Sleeman K.E.
      • Peruselli C.
      • Higginson I.J.
      Response and role of palliative care during the COVID-19 pandemic: a national telephone survey of hospices in Italy.
      ,
      • Zhang H.
      • Liu D.
      • Marks S.
      • et al.
      Usage patterns of a mobile palliative care application.
      ,
      • Vona P.
      • Wilmoth P.
      • Jaycox L.H.
      • et al.
      A Web-based platform to support an evidence-based mental health intervention: lessons from the CBITS Web Site.
      The number of pages per session refers to the number of webpages within the platform that the user viewed in a single session and the mean session duration referred to the mean duration of time that users spent on the platform (reported in minutes and seconds). This kind of user traffic data provide an approximation of the degree of exposure that users had to the content being hosted on the platform.
      • Costantini M.
      • Sleeman K.E.
      • Peruselli C.
      • Higginson I.J.
      Response and role of palliative care during the COVID-19 pandemic: a national telephone survey of hospices in Italy.
      Graphics demonstrating the geographic distribution of users were created by Google Analytics. Tables demonstrating the distribution of sessions, visits, most viewed pages, and device types were created using Microsoft Excel, version 16.36 (Microsoft Corporation, Seattle, WA).

      Results

      Characteristics of User Behavior

      There was a total of 2042 users and 4637 page views during the period of April 7, 2020, to May 1, 2020. Eighty-one percent of users were first-time visitors to the application, with the remaining 19% representing return visitors (Fig. 4). Seventy-five percent of sessions were accessed by first-time or second-time visitors to the application (Table 1). Users spent an average of 2 minutes and 6 seconds per session on PalliCOVID, with 77% of sessions lasting less than one minute (Table 2).
      Figure thumbnail gr4
      Fig. 4Proportion of new users vs. returning users.
      Table 1Proportion of Total Sessions Grouped by Number of Visits to PalliCOVID
      VisitsNumber of Sessions (%)
      11992 (61.2)
      2460 (14.1)
      3215 (6.6)
      4122 (3.8)
      5–8234 (7.2)
      9–14111 (3.4)
      15–2583 (2.6)
      26–5038 (1.2)
      Table 2Sessions Categorized by Duration of Time Spent on PalliCOVID
      Session DurationNumber of Sessions (%)
      ≤1 minute2499 (77)
      1–3 minutes324 (10)
      3–10 minutes234 (7)
      >10 minutes198 (6)

      Platform Improvement

      The most viewed page was the home page (96.5% of total page views), followed by Quick Guides to Symptom Management (an example of which is included as Fig. 1), the page with indexed links to COVIDprotocols.org, the section on Emergency Department resources, and the collection of videos simulating different types of difficult conversations (Table 3). Almost all users used either mobile devices (56%) or desktop computers (42%) to access the application (Table 4). Only 2% of users accessed PalliCOVID by tablet device. Most users (87%) accessed PalliCOVID from the United States, with 32% of all users located in Boston and the surrounding cities (Fig. 5). The remainder of users accessed PalliCOVID from 29 other countries outside of the United States.
      Table 3The Top Five Most Viewed Pages Based on Number of Page Views (N = 4637)
      PagePage Views (%)
      Home page4474 (96.5)
      Quick Guides to Symptom Management23 (0.5)
      COVIDprotocols.org22 (0.5)
      Emergency Department Resources20 (0.4)
      Video Simulations of Difficult Conversations16 (0.4)
      Table 4Type of Device Used to Access PalliCOVID
      DeviceNumber of Sessions (%)Pages Viewed per Session (n)Mean Session Duration
      Mobile1835 (56)1.441 minute, 49 seconds
      Desktop1355 (42)1.422 minutes, 30 seconds
      Tablet65 (2)1.141 minute, 29 seconds
      Figure thumbnail gr5
      Fig. 5Geographic distribution of user density by city.

      Discussion

      In the response to the large influx of seriously ill patients during the peak period of the COVID-19 pandemic in Boston, PalliCOVID was created as a scalable digital solution to increase access to palliative care educational resources and specialist support. Analyzing user engagement behavior, we found that three-quarters of sessions (75%) consisted of first-time and second-time visitors to the application. The remaining 25% of sessions represented usage by “frequent users,” defined as those who have visited the application three or more times. A previous study on usage patterns of a mobile palliative care application found that a small subset of users (13% of users) comprised the majority of all activities in the application (66% of activity).
      • Zhang H.
      • Liu D.
      • Marks S.
      • et al.
      Usage patterns of a mobile palliative care application.
      One possible explanation for this distribution is frequent users coming back to the application as a “just in time” clinical reference tool, whereas infrequent users access the application for more passive educational purposes.
      • Zhang H.
      • Liu D.
      • Marks S.
      • et al.
      Usage patterns of a mobile palliative care application.
      PalliCOVID content features both types of information: “just in time” clinical reference material such as medication dosing recommendations as well as focused educational content such as difficult conversation simulation videos, which may explain the different distribution of frequent vs. infrequent users. Furthermore, we limited our data-collection period to less than four weeks of operation, which may not be enough time to observe trends in users returning to the application.
      Average duration of sessions on PalliCOVID was short, at less than 2 minutes per session for mobile users and less than 3 minutes per session for desktop users. Some may interpret the short session duration to indicate a degree of user disengagement, but an alternative explanation may be that users were able to quickly find what they were looking for. Sessions in which users access the link to the hospital paging directory from the PalliCOVID home page, for example, would cause the user to exit the application and end the session relatively quickly. In that case, a short session duration would have indicated that PalliCOVID had successfully carried out a function that it was designed to perform.
      The platform can be improved based on user behavior. Future iterations of the application will focus on improving user engagement with the content pages for “Nursing Resources” and “Hospice Care Referrals”, which did not rank among the top five most-viewed pages. The lower ranking of these pages may be a reflection of the original dissemination strategy, which focused more heavily on reaching physicians and advanced practice providers rather than the nursing staff. Future dissemination strategies should specifically target nurses or have product design specifications to specific user populations as well. Population and user-specific design may be critical as different roles such as nursing spend the majority of their time at patients' bedsides and are more likely to be aware of patients' specific symptoms and end-of-life needs than perhaps physicians or advanced practice providers.
      There was an overall even split in device types between mobile and desktop, arguing against a stronger user preference for one type of device over the other. As we add future content to the platform, we should continue to optimize viewing for both smaller screens and larger screens with vector graphics, responsive Web design, and screen size flexibility.
      When observing the location of the users, 32% of users were found to be from Boston and surrounding cities. This finding aligns with our implementation science focus on engaging users about the platform primarily within the Partners institutions. In addition, a subset of the content was Partners institution specific, and paging directory access was restricted to members of the Partners network. Future reorganization of the content into internal- and external-facing sections may make the application more inviting to users outside of the Partners enterprise. The majority of the clinical content contained within the guides would be applicable to patient care regardless of the particular institution, so future improvements to the platform may include better sign-posting of content as broadly applicable and not Partners specific.

      Limitations

      Although Google Analytics has provided promising data about the usage patterns of PalliCOVID, the data should be interpreted cautiously. Google Analytics collects and presents data with a marketing and e-commerce framework for understanding Web-based behavior, rather than a health research frame. Aggregation of the data, in addition to potential discrepancies in how users are counted when users delete browser cookies, switch devices, or create a new client ID by changing browsers, creates bias in the data. Nonetheless, we believe there is still value in tracking overall engagement numbers and observing trends in usage over time to continually evaluate and improve the performance of the platform.
      As the volume of patients with COVID-19 infection rises and falls through different phases of the pandemic, the need for a digital solution such as PalliCOVID may also fluctuate over time.
      • Costantini M.
      • Sleeman K.E.
      • Peruselli C.
      • Higginson I.J.
      Response and role of palliative care during the COVID-19 pandemic: a national telephone survey of hospices in Italy.
      ,
      • Ferguson L.
      • Barham D.
      Palliative care pandemic pack: a specialist palliative care service response to planning the COVID-19 pandemic.
      Because our team focused our efforts on the development of a resource that would be maximally useful to clinicians within the Partners HealthCare System, the sample size was limited to its distribution within this institution and geographic area. We considered uptake of the resource outside of the Partners enterprise to be a secondary benefit. Future studies should include a more detailed analysis of longitudinal user data or a mixed-method assessment with focus groups, surveys, or in-depth interviews, to supplement Google Analytics data and allow for a more comprehensive quality-improvement study.

      Conclusion

      This study described usage patterns of a Web-based palliative care content platform over its first month of operation in April 2020. We have demonstrated that, even in the midst of a global pandemic, it is possible to rapidly design and implement a digital solution in response to an unprecedented health-care challenge. We have also demonstrated the use of a free, open-access tool such as Google Analytics to evaluate patterns of user behavior, consequences of the dissemination strategy, and aspects of the platform that may be amenable to future improvements. Quantitative data should be combined with qualitative research to provide more accurate interpretations of user behavior. We hope that clinicians who access the content on PalliCOVID may feel empowered to deliver care that is focused on dignity, symptom control, and avoidance of unnecessarily invasive or nonbeneficial interventions whenever possible.

      Disclosures and Acknowledgments

      This research received no specific funding/grant from any funding agency in the public, commercial, or not-for-profit sectors. The authors declare no conflicts of interest.

      References

        • Curtis J.R.
        • Kross E.K.
        • Stapleton R.D.
        The importance of addressing advance care planning and decisions about do-not-resuscitate orders during novel coronavirus 2019 (COVID-19).
        JAMA. 2020; https://doi.org/10.1001/jama.2020.4894
        • Fausto J.
        • Hirano L.
        • Lam D.
        • et al.
        Creating a palliative care inpatient response plan for COVID19 - the UW medicine experience.
        J Pain Symptom Manage. 2020; 60: e21-e26
        • Heyland D.K.
        What matters most in end-of-life care: perceptions of seriously ill patients and their family members.
        CMAJ. 2006; 174: 627-633
        • Goldstein J.
        • Rothfeld M.
        • Weiser B.
        Patient has virus and serious cancer. Should doctors withhold ventilator? The New York times.
        (Availble from)
        • Emanuel E.J.
        • Persad G.
        • Upshur R.
        • et al.
        Fair allocation of scarce medical resources in the time of Covid-19.
        N Engl J Med. 2020; 382: 2049-2055
        • Arya A.
        • Buchman S.
        • Gagnon B.
        • Downar J.
        Pandemic palliative care: beyond ventilators and saving lives.
        CMAJ. 2020; 192: E400-E404
        • Stone S.C.
        • Mohanty S.
        • Grudzen C.R.
        • et al.
        Emergency medicine physicians' perspectives of providing palliative care in an emergency department.
        J Palliat Med. 2011; 14: 1333-1338
        • Rêgo F.
        • Portela F.
        • Santos M.F.
        Towards PWA in healthcare.
        Proced Computer Sci. 2019; 160: 678-683
        • Hawley P.
        Barriers to access to palliative care.
        Palliat Care Res Treat. 2017; 10 (1178224216688887)
        • Costantini M.
        • Sleeman K.E.
        • Peruselli C.
        • Higginson I.J.
        Response and role of palliative care during the COVID-19 pandemic: a national telephone survey of hospices in Italy.
        Palliat Med. 2020; 34: 889-895
        • Ogrinc G.
        • Davies L.
        • Goodman D.
        • Batalden P.
        • Davidoff F.
        • Stevens D.
        SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process: Table 1.
        BMJ Qual Saf. 2016; 25: 986-992
        • Song M.J.
        • Ward J.
        • Choi F.
        • et al.
        A process evaluation of a web-based mental health portal (WalkAlong) using Google Analytics.
        JMIR Ment Health. 2018; 5: e50
        • Clark D.J.
        • Nicholas D.
        • Jamali H.R.
        Evaluating information seeking and use in the changing virtual world: the emerging role of Google Analytics.
        Learn Pub. 2014; 27: 185-194
        • Kirk M.
        • Morgan R.
        • Tonkin E.
        • McDonald K.
        • Skirton H.
        An objective approach to evaluating an internet-delivered genetics education resource developed for nurses: using Google AnalyticsTM to monitor global visitor engagement.
        J Res Nurs. 2012; 17: 557-579
        • Zhang H.
        • Liu D.
        • Marks S.
        • et al.
        Usage patterns of a mobile palliative care application.
        J Palliat Med. 2018; 21: 796-801
        • Vona P.
        • Wilmoth P.
        • Jaycox L.H.
        • et al.
        A Web-based platform to support an evidence-based mental health intervention: lessons from the CBITS Web Site.
        Psychiatr Serv. 2014; 65: 1381-1384
        • Ferguson L.
        • Barham D.
        Palliative care pandemic pack: a specialist palliative care service response to planning the COVID-19 pandemic.
        J Pain Symptom Manage. 2020; 60: e18-e20