Automated contact tracing assessment
November 17, 2022
MIT Lincoln Laboratory Report TR-1287
The COVID-19 pandemic placed unprecedented demands on the global public health systems for disease surveillance and contact tracing. Engineers and scientists recognized that it might be possible to augment the efforts of public health teams, if a system for automated digital contact tracing could be quickly devised and deployed to the population of smartphones. The Private Automated Contact Tracing (PACT) protocol was one of several digital contact tracing proposals offered worldwide. PACT’s mission—to preserve individuals’ privacy and anonymity while enabling them to quickly alert even nearby strangers of a likely risky exposure—was adopted by Google and Apple and realized in the Exposure Notifications (EN) service and API for mobile application development. The Exposure Notifications system, like many digital proximity tools, is based on Bluetooth signal strength estimation, and keeps much of the necessary information and computation on the smartphones themselves. It implemented a decentralized approach to contact tracing: the public health authority, and other governmental authorities, cannot access the records of an individual’s encounters with others; nor is physical location used or shared by the service. Although the service is available on most modern iOS and Android devices, it is not enabled by default; the individual must opt in to use a particular region’s implementation of the service, either by installing the regional app or by enrolling through a menu of regions in the operating system settings. Likewise, individuals must affirm their consent before the service can share anonymized infection status with the regional public health authority, and alert recent close contacts. The widespread availability of Exposure Notifications through Apple and Google’s platforms has made it a de facto world standard. Determining its accuracy and effectiveness as a public health tool has been a subject of intense interest. In July 2020, CDC’s Innovative Technologies Team designated MIT LL and the PACT team as trusted technical advisors on the deployment of private automated contact tracing systems as part of its overall public health response to COVID-19. The Innovative Technologies Team sought to answer the following key question regarding automated contact tracing: Does automated contact tracing have sufficient public health value that it is worthwhile to integrate it at scale into existing and evolving manual contact tracing systems? Rapidly rising caseloads necessitated parallel-path assessment activities of most mature systems at the time. When access to the Google and Apple Exposure Notifications system became available, MIT LL focused the assessment efforts on the systems being built and deployed. There were two immediate and significant challenges to observing and quantifying the performance of the system as a whole: first, the privacy preserving design decisions of PACT and the system implementers denied access to system-level performance metrics, and second, obtaining accurate “ground truth” data about risky encounters in the population, against which to measure the detector performance, would require an unacceptable level of effort and intrusion. Therefore, MIT LL designed a set of parallel research activities to decompose the problem into components that could be assessed quantifiably (Bluetooth sensor performance, algorithm performance, user preferences and behaviors), components that could be assessed qualitatively (potential cybersecurity risks, potential for malicious use), and components that could be modeled based on current and emergent knowledge (population-level effects). The MIT LL research team conducted early assessments of the privacy and security aspects of new EN app implementations and closely reviewed the available system code exercised by the apps, before conducting a series of phone-to-phone data collections both in the laboratory and in simulated real-world conditions. The data from these experiments fed into models and visualization tools created to predict and understand the risk score output of candidate “weights and thresholds” configurations for EN, i.e., to predict the performance of the system as-built against ground truth data for distance and duration of “exposure”. The data and performance predictions from this effort helped to inform the global and local community of practice in making configuration decisions, and can help to predict the performance of future versions of similar tools, or alternative implementations of the current system. We conducted a human factors and usability review of early app user interfaces and messaging from public health, and designed a follow-on large-scale survey to investigate questions about user trust and system adoption decisions. The results of the human factors, user trust, and adoption studies were used by U.S. public health jurisdictions to make adjustments to public-facing communications, and were shared with Apple and Google to improve the user interface. Information gathered from public health experts enabled us to better understand conventional contact tracing workflows and data streams, and we incorporated that information into an agent-based model of “hybrid” contact tracing plus Exposure Notifications. We then combined it with emerging reports on vaccination, mask effectiveness, social interaction, variant transmissibility, and our own data on the sensitivity and specificity of the Bluetooth “dose” estimator, to predict system-level effects under various conditions. Finally, we helped to establish a network of Exposure Notifications “practitioners” in public health, who surfaced desirable system-level key performance indicators (implemented during 2021 and 2022, in the Exposure Notifications Private Analytics system, or ENPA). At the conclusion of the program, many of the initial conditions of the pandemic had changed. The Exposure Notifications service was available to most of the world, but had only been deployed by 28 U.S. states and territories, and had not been adopted by much of the population in those regions. High case rates during the Omicron surge (December 2021 – January 2022) and newly available ENPA data offered the first hints at calculating “real” state-level performance metrics, but those data belong to the states and many are cautious about publishing. Although Google and Apple have stated that Exposure Notifications was designed for COVID-19, and will not be maintained in its current form after the pandemic ends, the public health and engineering communities show clear interest in using the “lessons learned” from Exposure Notifications and other similar solutions to preserve the capabilities developed and prepare better systems for future public health emergencies. The intent of this report is to document the work that has been completed, as well as to inform where the work could be updated or adapted to meet future needs.