Could real-world evidence help to accelerate drug and device testing for COVID-19?

As companies seek to begin testing medical products to treat, cure or prevent COVID-19, they are seeking to collect clinical evidence faster than ever before. While Real-World Evidence could help to expedite the collection of evidence, there are several major impediments to the collection of high-quality data. AgencyIQ explains the promise of—and hurdles to—RWE for COVID-19.


To contact the author of this analysis, please email Laura DiAngelo.
To contact the editor of this analysis, please email Alec Gaffney.


Executive IQ Brief

  • How Things Work Now: Most data to support the approval of a new medical product is obtained from randomized, controlled clinical trials. However, there exists significant evidence from electronic health records and other data sources on the actual use of approved products that could support additional approval decisions, such as a new indication of use for a drug, or follow-up testing for an approved product. This type of evidence is known as Real-World Evidence (RWE).
  • What’s New: As the US confronts COVID-19, many companies are now interested in ways to accelerate the testing of medical products. One method would be to test already-approved products, and to look at outcomes data from patients to determine if it works. However, there are many limitations to the use of this data, including a lack of standardization and regulatory support.
  • What’s Next: Any reforms to the systems which could be used to collect RWE are unlikely to occur prior to the end of the COVID-19 outbreak. The use of RWE is therefore likely to be fragmented and depend on existing systems, as imperfect as they are. While the FDA is likely to make use of RWE to make regulatory decisions, it may also require data to be supported by other pre- or post-market data.

Regulatory Background

Under the Federal Food, Drug, and Cosmetic (FD&C) Act, approval of a drug or biologic by the FDA requires that the product demonstrate “substantial evidence” that it is safe and effective for use. The FDA has traditionally interpreted the term “substantial evidence” as two well-controlled clinical trials, although it often allows a single trial to provide “substantial” evidence when multiple trials would be infeasible (such as for rare diseases with small populations or hard-to-treat and quickly debilitating diseases).


[AgencyIQ subscribers can read analysis of the changing evidence standards here.]

While randomized, controlled clinical trials (RCTs) remain the gold standard for the generation of evidence, there are other forms of evidence that also inform regulatory decision making. This “real world data” (RWD) can be collected and analyzed to generate “real world evidence” (RWE).


FDA defines RWD as “the data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources,” including insurance and medical claims, electronic health or medical records (EHR/EMR), and disease or clinical registries. It defines RWE as “the clinical evidence regarding the usage and potential benefits or risks of a medical product derived from analysis of RWD.”


The basic idea of using RWE is simple: Once approved, many products are used both in accordance with their label and “off-label.” Some products many be more useful than initially thought, while others may have other uses that may not have been studied. Data about these uses could help a company to expand their approved label without needing to conduct extensive clinical testing.


A key source of potential RWE is data gathered during the course of medical treatment through an EHR or EMR. However, the validation and collection of that data represents significant operational and analytical hurdles.


As noted by many FDA senior staffers in a 2016 piece published in the New England Journal of Medicine, one of the most critical challenges is data validation. A database with poor-quality data likely will lead to poor-quality conclusions. Studies should also incorporate concepts from existing clinical trials, such as randomization, to minimize intentional or unintentional bias.

However, the control of some elements are inherently difficult. RCTs are well-controlled and closely regulated, with specific protocols for everything from the researcher’s language when surveying patients to clinical researcher’s clothing around samples. These types of factors are not generally controllable outside of the trial environment.


Regulatory Context

There are more than 140 therapeutic medical products currently in development to respond to COVID-19, including 24 drugs and biologics that have already been approved for other indications that may be effective against the virus. Of the 140 products, at least 40 are already in clinical development.

[Click here to download a copy of AgencyIQ’s COVID-19 development tracker.]


The FDA is under intense pressure from the White House, Congress, and the American public to clear or authorize products that can treat or prevent COVID-19 as quickly as possible. As a result, it is likely to do using limited evidence or non-traditional evidence. One type of evidence that might support approval decision-making is RWE.


This approach might be especially useful for products that have already been approved for other uses and are being used off-label for COVID-19.


Already, the Patient Centered Outcome Research Network (PCORNet), in a partnership with Duke, has initiated an RWE-based pragmatic, randomized trial, using a smartphone-based enrollment system for willing health care workers to identify if hydroxychloroquine has a prophylactic effect on the virus. While the intent of the study, the Healthcare Worker Exposure Response and Outcomes (HERO) Registry and Hydroxychlorquine (HERO-HCQ) Trial Program, is limited to the prevention of COVID-19 in health care workers, the results are likely to inform policy at a national level.


For unapproved products cleared for use in responding to COVID-19, RWE generation and validation are likely to be a key part of post-market follow up. Gilead’s remdesivir, for example, is currently in clinical testing as a potential treatment for the virus. However, any data presented to the FDA as a result of those trials will be limited to the population studied, rather than the whole of the US population writ large. In regular circumstances, the FDA will only approve a drug for a population if there is clinical evidence that the drug is effective and safe for that specific population.


However, any drug approved for COVID-19 is likely to be used beyond its approved label by healthcare professionals. RWE-generating studies looking at the effects of these off-label uses in difference age, gender and racial groups could help further expand (or restrict) the product’s label.


RWE Data Sources: Current and Potential

The US’ response to COVID-19 is resulting in an unprecedented amount of data that could potentially be leveraged by the life sciences industry to help respond to, or prevent, infections. However, it highlights a significant question central to the ongoing regulatory debate about the use of RWE: Are the data fit to purpose?


There are many sources of data that might be leveraged by industry to advance the development of medical products to treat COVID-19. Those sources, and their limitations, are explained below.



In December of 2019, the FDA launched a mobile app called CURE ID to support RWE generation. The CURE ID app is intended to be used to allow prescribers to “report their experiences treating difficult-to-treat infections diseases with novel uses of existing FDA-approved drugs.”


CURE ID can be accessed via a smart phone, tablet, website or other mobile device, and can serve as a repository for off-label use reporting. According to FDA Principal Deputy Commissioner Amy Abernathy, “CURE ID allows… real-world experiences to be organized and analyzed much faster, making it easier to spot promising new uses for existing drugs.”


The application is broadly understood as the FDA’s attempt to establish a unified repository of off-label uses of already approved drugs that could help more quickly identify treatments for infectious diseases. While the White House has heralded such practices in treatments for COVID-19, however, there is no evidence that the application is being used as intended in this outbreak. 


While an FDA spokesperson told IDSE that “COVID-19 cases have been submitted by clinicians to the CURE ID platform,” the app is not being leveraged as a significant repository of experience or understanding about off-label use, and there are no specifics available on how many clinicians are leveraging the website.


However, it’s unclear if the tool is inherently not able to function as intended, or if the timing for CURE ID’s launch worked against it for the current outbreak. The platform was launched in early December 2019, giving providers limited time to download the application, understand it, and work it into their clinical workflow before being overrun with COVID-19 cases. As the use of CURE ID by clinicians is currently voluntary, it would be considered a supplement to other types of reporting—including regular public health reporting—and, at the moment, would likely place additional reporting burden on providers. In future outbreaks, CURE ID could be a very useful tool to track and generate data on off-label use of approved products.


EMR/EHR Linking

Potentially the most significant source of RWD are individual medical records, which capture patient demographics, therapy use and outcomes for individual patients over time. In recent years, there has been a significant push to digitize medical records, leading to the widespread use of EHRs and EMRs.


In 2011, the Centers for Medicare and Medicaid Services (CMS) implemented a program called the Medicare EHR Incentive Program, more commonly known as “Meaningful Use” (MU). Under this program, providers were required to use Certified EHR Technology, or CEHRT, in order to qualify for certain payments. Although the program has been updated significantly since its implementation—it is now called “Advancing Care Information”—the program and its requirements still exist for providers that accept payments from Medicare.


The FDA’s regulatory authority over electronic records is quite limited, with the FDA interpreting its authority narrowly “to permit the widest possible use of electronic technology.” Instead, authority over the certification of EHR technology largely belongs to the HHS Office of the National Coordinator for Health Information Technology (ONC), which sets standards for the development and capabilities that an EHR technology must have in order to be certified, and therefore eligible for use by entities seeking full payments under Medicare.


The basic tenet of HIT data sharing, or “interoperability,” is that all the products need to speak the same language (i.e., use the same coordinated data standards). Standardization of these data elements is one of the main tasks of ONC.


COVID Coding

In order for clinical data to be shared, it needs to be coded in a way that is unified across EHRs. This is a challenge in the best of circumstances. However, there are not yet any specifically established standards for COVID-19. Although CMS announced that it is making COVID-19 codes available in the Medicare and Medicaid systems, this may not be operationalized as planned. 


According to the discussions at a March 26 meeting of the Health Information Technology Advisory Committee (HITAC), a nonpartisan research and advisory group that provides recommendations to ONC, those codes may not be operational by the target date. These challenges were outlined by HITAC Commissioner Aaron Miri at the meeting, who stated that “there are a number electronic medical vendors out there that are reluctant to add this code in anytime soon. At lot of the EMR vendors have put a pause on any new updates, therefore we cannot code using this new ICD-10 code.”


A lag in rolling out the electronic terminology that would allow for activities such as bio surveillance, disease surveillance, and other activities that would need to occur for the generation of RWD could limit a manufacturer or data vendor’s ability to identify and validate data from EHRs or EMRs. As HITAC Commissioner Arien Malec noted at the meeting, “We have a huge amount of data that’s sitting in the EHRs right now that could be used for looking at disease surveillance and additional risk factors, and that data is not being as accessible as it could be,” due to a lack of certain foundational activities like consistent data terminology.


This means that although patients are being seen by providers using EHRs, and presumably data is being captured there about their health status, treatment protocol and response—as it would be in a clinical trial—it may be impossible to link this data into a complete picture that would be considered validated RWD.


Data Linking

One potential way for EHR data to be collected and validated for patients with certain conditions could be linking EHRs to clinical registries, creating a more comprehensive picture of the patient’s journey through the health care system, their treatments and their outcomes. Notably, under the 21st Century Cures Act, the ONC was directed to establish a framework for the bidirectional data exchange between EHRs and clinical registries. The intent behind the provision was to increase the availability of health care data to researchers, which would help support the generation of RWE.


However, that provision has yet to be implemented broadly. The certification criteria by which ONC oversees standardized EHRs (which is called the “2015 Edition”) includes specific cases for bidirectional exchange, including Immunization Information Systems. However, the exact registry provision is not yet operationalized—although the Office did requested comment on how that should work on a proposed rule in March of last year (Section 4005). To date, though, the agency has not formally proposed (or finalized) any provision that would broadly require the implementation of these capabilities into EHRs.


Each source of RWD—EMR/EHR data, claims data, patient-reported outcomes, digital apps—provide a single snapshot into a disease or condition and potential treatment efficacy. In order to translate these data into evidence, they must be combined into a single repository and validated. That is, a patient’s medical records from an EHR must match the identifiers of the same patient in the clinical registry and public health reporting and should align with data captured from mobile apps or patient-reported outcomes.


This practice, called “patient matching,” has been a key challenge for interoperability for several years, according to Ben Moscovitch, Director of the Health Information Technology Project at the Pew Charitable Trusts. In an interview with AgencyIQ, Moscovitch explained that “inadequate patient identification and patient matching” leads to difficulties utilizing the data.


For example, if a patient’s address is written differently in their EHR and in a registry, the data will not be able to be linked, creating a blind spot that would invalidate the data. While “ONC has taken some steps to advance patient matching among EHRs,” Moscovitch said, “additional steps are also needed.” For example, as part of the USCDI (the data element standard recently adopted for use by Certified EHRs), “ONC added email addresses and previous address to those data elements that should be made available and could be used for matching. Those data elements have not been used robustly for matching and improve match rates.”


However, that policy will only increase EHR interoperability with another Certified EHR—not with other sources of data, such as a clinical registry. In looking forward to ensuring cross-source data exchange, Moscovitch stated that “ONC can also serve in a coordinating role to further expand patient matching principles beyond Certified products. For example, as registries require additional data matching, ONC can help ensure that these different systems are communicating in a robust way.”


ONC has worked to adopt standards and capabilities in EHR policy that would ensure feasible functionality in a practice setting. There is one priority use-case example of bidirectional exchange that has already been implemented and is being tested between providers, states and registries: Prescription Drug monitoring Programs (PDMPs), or tools that are meant to identify and track individuals at-risk of developing opioid use disorder (OUD). However, difficulties getting these platforms up, running, and functional indicate that there is still significant work to be done on this front before it can be reliably used for data validation.


Proposed Surveillance System

With a population-wide spread, the outbreak of COVID-19 could represent a significant opportunity to leverage RWD in evidence generation. This idea was recently put forth by former FDA Commissioners Scott Gottlieb and Mark McClellan in a paper published by the Duke Margolis Center for Health Policy, of which McClellan currently serves as Director. That paper argues the US “can support large scale access to different drugs that have shown they may be effective against the coronavirus in a framework that enables us to collect good information to determine which medicines are working best for patients, and ultimately merit full FDA approval.”


In a subsequent paper issued on April 7th, Gottlieb and McClellan called for the establishment of a robust national surveillance system to respond to the outbreak of COVID-19. This system, known as the National Syndromic Surveillance System, would be coordinated by the CDC and have four high-priority capabilities:

  1. establish a widespread diagnostic testing and data sharing infrastructure;
  2. integrate local test and trace infrastructures into an enhanced national surveillance system;
  3. conduct widespread serological testing to identify reliable markers of immunity; and
  4. establish a rapid response capability for identified cases.

That system would leverage payment data from health plans, including the federal Medicare and Medicaid programs, to support surveillance and response. For example, they state that CMS could provide guidance offering additional payment incentives for test platforms with effective electronic data sharing that have high quality and interoperable (i.e., transmittable and using common data elements that can be ready by other systems) data.


Under this plan, the Office of the National Coordinator for Health IT (ONC) would “encourage” the adoption of electronic standards and reporting measures for COVID-19, which could help to link together data about patients with COVID-19. However, this plan does not yet exist, and the logistical challenges of implementing such a system are considerable.


Vaccine Registries

As noted by lawmakers, public health officials, and highlighted in Gottlieb and McClellan’s paper, the development of—and ability to track—a vaccine and immunity to COVID-19 will be a fundamental next step in the response to COVID-19.


While some lawmakers, including Senator Bill Cassidy (R-LA) have called for a new “immunity registry,” this is a capability that vaccine registries already have, according to the Executive Director of the American Immunization Registry Association, Rebecca Coyle. Currently, 49 states have a vaccine registry, and many are set up to support bidirectional data exchange.


According to Coyle, the vaccine registry system has been improved drastically since its most recent pre-COVID pressure test, the H1N1 outbreak. “The most significant changes we’ve seen [are] how data flows in this country,” Coyle told AgencyIQ in an interview. “Pre-H1N1, it was all paper-based records or user interface on the web,” and there were gaps in the registry system’s ability to track adult immunization since most vaccinations happen to children and adolescents.


After H1N1, the registry system in the US took action to address gaps in functionality, including integrating adult reporting into a system originally designed to capture pediatric and adolescent information. Currently, most large provider health systems are connected to at least one registry, which helps support patient matching between the EHR and the registry.


The vaccine registry system represents a unique opportunity for data collection. Typically, vaccines are both ordered and recorded through the registry in most states, and registries include an inventory management component. For publicly supplied vaccines, CDC reviews orders before they are filled, allowing for a systemic tracking. If the vaccine system for COVID-19 follows the same path as that in place for the H1N1 outbreak, the CDC will—once a vaccine is available—set up a commercial ordering and distribution center.


However, like other potential sources of RWE, vaccine registries were not originally established to support drug development. There are some gaps that could preclude the usefulness of the data for the FDA. For example, any adverse events associated with a vaccine are required to be submitted to the CDC via the Vaccine Adverse Event Reporting System (VAERS), a joint venture with the FDA. However, that system is an online reporting tool that is not set up to electronically receive data—a provider must manually enter the output of their EHR or registry information into a web portal from the CDC. While this issue was recently highlighted by the National Vaccine Advisory Committee, the plan to modernize the system was never fully operationalized, says Coyle. Although the program is, according to its website, intended to “serve as a monitoring system in public health emergencies” and “assess the safety of newly licensed vaccines,” the manual difficulty of entering information in via VAERS could limit its use.


As already seen through the lack of Case Report forms for some COVID-confirmed cases, providers may be feeling especially burdened and have limited time for additional paperwork, creating a break in the data flow and potentially invalidating the usefulness of the data for the FDA. However, difficulties with manual data entry are already apparent. According to the April 3 CDC Morbidity and Mortality Weekly Report (MMWR), there were 122,653 cases of laboratory-confirmed COVID-19 reported to CDC by March 31. Of those, only 60.7% were accompanied by a case report form, a manual reporting system from the CDC


Monitoring of a potential vaccine in real time would serve two purposes for COVID-19 response. First, identifying who has immunity to the virus, either through vaccination or capturing individuals with diagnostic-confirmed COVID, could help better target testing and intervention capacity. Second, it could help track the performance of the vaccine to determine its effectiveness at preventing future cases of COVID-19.


Although vaccine registries do have a significant amount of data collection and linking capabilities, there are several key challenges to expanding their use. Primarily, the growth of the vaccine hesitancy (or “Antivax”) movement has resulted in different standards for data collection state by state. Some members of the public have significant distrust in immunization and vaccination processes and have taken action at the state level to limit the amount of information collected or where it can be shared. From a political standpoint, it may be difficult to establish a national surveillance program using the vaccine registries as a base infrastructure, given the levels of vaccine hesitancy.


What’s Next

As public health surveillance is conducted on a national level, and distribution of products intended to diagnose, treat, or prevent COVID-19 happens population-wide over the next few months, the opportunities to leverage that data for a regulatory purpose are likely to be challenging.


Drs. Gottlieb and McClellan state in the opening of their paper that the generation of data from “large scale access to different drugs” that “may be effective” against COVID-19 needs to have a framework that would enable data collection. At that time, it doesn’t appear that that framework 

exists in any sort of widespread capacity.


Life sciences companies are interested in leveraging RWE to support new indications for use, or to track the efficacy of authorized (or approved) treatments, but much of this evidence likely will be collected within distinct medical systems and not across the entire US. That could limit the broad applicability of the collected evidence. As a result, the FDA seems likely to require certain pre- or post-market obligations to validate that data.


As the FDA and US look to a post-COVID-19 reality, it’s possible—even probable—that RWE will become a core component of a reformed system. However, such reforms would take years to implement, even for ones with broad support and a lack of opposition.


Another key challenge is that drug development has not historically been a priority for HIT policy developers or public health surveillance system administrators. For example, priority areas for ONC identified at the recent HITAC meeting to target COVID-19 were the standardization of data elements that would be needed to build algorithms tied to more specifically actionable patient care, such as bed availability as an application program interface (API). Drug development wasn’t mentioned.


However, ONC could play a significant role in standardizing real-world data that is sourced from EHRs, which in turn could be used to support clinical trial information and real-world natural history trials that are sourced from EHR data. If underlying source data doesn’t have the associated standards that allow for case tracking, little of the information would be useful to the FDA when making a regulatory decision.


Public Testing

From a public health perspective, however, there is significant need for RWE generation in the next phase of pandemic response.


The White House and other governmental entities have been highlighting the need to “reopen” businesses, the economy, and workplaces. A hallmark of any plan to reopen businesses would be the requirement that individuals who have already had, and recovered, from the virus would be identified (either through symptomatic infection or serological testing) and allowed to return to regular activities on an expedited timeline.


“I think a lot of diagnostic companies would like to see predictive models to guide treatments,” McClellan told AgencyIQ in an interview. “If you combine the test result data with other information—maybe testing over time—does that help predict outcomes for patients? Those are great RWE questions. There’s lot more to do there.”


While the FDA has authorized serological tests for identifying antibodies associated with an immunological response to the virus, these tests aren’t validated based on the identification of antibody titration. In short, the tests should be able to identify whether an individual has previously had COVID-19, but there is not yet adequate research or science to demonstrate that a specific level of measured antibody presence would yield practical immunity—including the level of immunity, the duration of immunity, and the potential for relapse.


Ideally, the FDA should offer guidance on when and how to validate the reliability of serological testing to support evidence of practical immunity. In order to be operationally feasible, that system would need to be reliable with significant evidentiary backing. It will likely require integration of data from CDC, EMR/EHRs, and public health surveillance systems within the states—but will require FDA guidance and support.


A Focus for FDA


In general, while the FDA is generally willing to accept RWE, it will often require a company to augment that data with an RCT to validate its findings within a closely-controlled environment. Under the 21st Century Cures Act of 2016, the FDA was directed to increase the types of uses for which RWE is acceptable in regulatory decision-making. In addition, under the Prescription Drug User Fee Act (PDUFA)’s sixth iteration, the agency committed to “robust policy development” on RWE on how it could leverage this type of data for therapeutics.


The FDA’s policy framework on RWE has yet to be established. Industry has long sought increased use of RWE by the FDA. For example, in comments on a potential second version of the 21st Century Cures Act from late 2019, the trade organization PhRMA specifically called on policymakers to “prioritiz[e] evolving the regulatory framework to allow biopharmaceutical companies to harness the full potential of digital [research and development] approaches,” especially for clinical trial design, operation, and data collection and to allow RWE “help satisfy or meet the ‘substantial evidence’ requirement.”


As the FDA now confronts COVID-19, this effort may once again receive legislative attention.


Leave a Comment

Your email address will not be published. Required fields are marked *