PHE Expert Shafrin: Make Real-World Evidence the New Gold Standard
In an in-depth commentary for Pharmaceutical Market Europe, PHE Senior Economist Jason Shafrin argues for real-world evidence (RWE) to become the “gold standard” of scientific evidence and heath technology assessments.
Will Randomized Controlled Trials Become a Thing of the Past?
By Jason Shafrin, PhD
(as published in Pharmaceutical Marketing Europe)
Randomized controlled trials (RCTs) are regarded as the gold standard of scientific evidence, and for good reason. By randomizing a treatment across study arms, RCTs eliminate patient-treatment selection bias, resulting in reliable causal inference. In contrast, in the real world patients who are sicker may be more (or less) likely to receive certain treatments. Because treatments are given selectively, the true causal effect of a treatment on patient health in most real-world studies cannot be determined. Because of the perceived quality of clinical trials data, RCT data are used in most health technology assessment (HTA) appraisals around the world; in 2015, 90% of CADTH appraisals, 80% of PBAC appraisals, 64% of NICE appraisals and all IQWiG appraisals relied on RCT-based evidence.
Nevertheless, in recent years there has been a call to increase the use of real-world evidence (RWE). In the United States, the FDA this summer released a guidance document on the use of RWE to support regulatory decision-making for medical devices. The FDA already uses real-world data to measure pharmaceutical safety as part of its Sentinel Initiative. Further, the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) recommends the use of RWE to support coverage and payment decisions.
In this article, I examine some of the limitations of RCTs, discuss appropriate opportunities to use real-world evidence, and describe a path forward for adding reliable RWE analyses to our scientific tool kit.
Limitations of randomized controlled trials
Although RCTs are scientifically credible and feasible to conduct in the majority of cases, they have at least 3 appreciable shortcomings: scientific limitations, practicality, and cost. Scientific limitations start with the RCT patient population itself, which may be restrictive and not representative of the patient population likely to use a treatment in the real world. In addition, health care practices participating in RCTs treat patients according to detailed protocols, making the care received in clinical trials unlikely to reflect what would be received in the real world. Furthermore, the outcomes of interest of an RCT may not be those of interest to patients. For instance, oncology trials often use surrogate outcomes, such as progression-free survival, rather than overall survival to measure efficacy. Research has shown that surrogate outcomes such as PFS have real-world survival gains that are generally 16% less than what is predicted by clinical trials. Traditionally, few RCTs would collect patient-reported outcomes, although that is beginning to change.
Even if these scientific limitations can be addressed, in some cases use of an RCT is simply not practical. For ultra-rare diseases, for instance, it may be infeasible to recruit a sufficient number of patients to randomize across multiple treatment arms. Consider also the case of a new digital medicine formulated as a chip-embedded pill to monitor adherence. A potential RCT could compare adherence levels and outcomes for patients using this digital medicine against standard of care (SOC). Within a clinical trial setting, however, the mandated additional physician visits and structured clinical protocols would likely result in the SOC arm appearing to be more effective than is the case in the real world; thus, the benefits of remote monitoring of adherence would likely be underestimated within an RCT setting.
RCTs are, of course, costly. The capital needed to bring a drug to market is now estimated to be $2.6 billion, with a large share of this cost related to RCTs. Cost has increased over time. According to the Independent Institute, the typical drug in 1980 underwent 30 clinical trials involving about 1500 patients; by the mid-1990s, the typical drug was subjected to more than 60 clinical trials involving nearly 5000 patients. In addition, securing regulatory approval for a drug can take decades. Given limited healthcare budgets, it is not feasible to conduct an RCT for every clinical scenario.
The promise and limitations of real-world evidence
While conventional wisdom holds that RWE is not appropriate to measure treatment efficacy for an initial indication for the vast majority of treatments, there are clear cases where RWE would be highly useful. In cases of ultra-rare diseases, RWE can play a pivotal role in approval. Additionally, phase 4 trials using RWE can help stakeholders understand a treatment’s long-term efficacy and safety. RWE potentially could also be used to evaluate effectiveness and safety of an already approved treatment for a new indication. More than 1 in 5 prescriptions in the United States is for an off-label use; providing additional evidence to better understand treatment efficacy and safety is crucial.
A key limitation of any study using RWE is the inability to statistically identify whether the treatment of interest was the sole factor affecting patient outcomes. However, randomization can be incorporated into real-world study designs. For instance, pragmatic trial designs blend randomization with real-world treatment practices. Another valid approach to assess treatment efficacy and safety in a real-world setting is cluster randomization, which randomizes provider sites by intervention or current SOC.
A proposal from Anirban Basu would even use RWE to measure treatment efficacy for granting an initial indication. Dr. Basu supports using phase 2 clinical trials to determine drug safety and then replacing phase 3 clinical trials with only-in-research (OIR) labeling over a specified period (eg, 2 years), using RWE to measure efficacy. During the proposed OIR period, only half of adults would be eligible to receive the treatment. Manufacturers would then prepare a protocol to conduct real-world studies on the incremental effectiveness of the use of the treatment relative to the standard of care. By randomizing access to the treatment, researchers would be able to measure efficacy using robust statistical methods. While intriguing, the real-world–based OIR process is not currently under regulatory consideration.
A path forward
Real-world evidence is already in use around the world. In the United Kingdom, NICE tends to support economic arguments in the postregulatory setting. France uses RWE as part of postmarketing evaluations, for innovative drugs requiring 5-year reevaluation of pricing and reimbursement. The FDA uses RWE to monitor drug safety as part of the Sentinel project. Additionally, RWE is being used to inform creative drug pricing. When seeking coverage for Kymriah, a new CAR-T therapy, Novartis has an agreement with the Centers for Medicaid and Medicare Services to adjust the price of the treatment based on the health outcomes of real-world patients.
Scientific standards, particular around transparency, need to be put in place to assure the scientific rigor of any RWE study. For starters, researchers should make available study protocols for all research using real-world data. Posting study protocols online can help prevent data fishing, a phenomenon whereby researchers conduct multiple data analyses and publish only what they determine to be favorable results. Additionally, to the greatest extent possible, study code should be made publicly available online. Use of RWE evidence often requires the use of advanced statistical techniques, such as propensity score matching, instrumental variables, and regression discontinuity. Because of the complexities of these statistical approaches, making the underlying real-world data analysis public will be critical for building the credibility of using RWE.
Real-world data analysis is only useful in the presence of high-quality real-world data. Policymakers, industry, patient advocates, and others should work together to develop new sources of high-quality real-world data. The creation of robust, detailed patient registries is one attractive data source. Health insurance claims and electronic medical records represent other high-quality sources of data. As healthcare becomes increasingly integrated into patient’s mobile phones, data from remote monitoring can also be useful. Even social media data can be used to measure real-world treatment safety and effectiveness.
In short, RWE presents a fantastic opportunity to accelerate drug approval, monitor treatment safety, and appropriately price medications. Realizing the opportunities, however, will require investment in new data sources as well as coordination across stakeholders to make RWE study methods more transparent.
By: Jason Shafrin, PhD Sr. Director, Policy and Economics and Jacki Chou, MPL Senior Director, Policy and Economics The recent announcement that CVS Caremark will use value-based metrics to inform formulary design at first appears a step in the right direction. However, using value measurements from a single nongovernmental organization—the Institute for Clinical and Economic Review (ICER)—based on a limited picture of treatment value may be problematic. Using this single assessment of value in an effort solely to exclude drugs from a major insurance formulary is even more worrisome and fails to recognize the heterogeneity in patient clinical needs and preferences.