The 21st Century Cures Act spurs the idea of “real-world evidence” in clinical research. This act brings a compelling prospect: that the drug and health companies can include data from electronic health records, claims data and other data sources to augment the more time consuming and expensive gold standard of controlled, interventional clinical trials. Recently, New York based Aetion published their real-world health study findings in an attempt to replicate the results of a randomized, controlled clinical trial.

TrialSite News provides a breakdown of the Aetion study results and some commentary of some potential limitations to this approach.  We reviewed other reports and articles relating to the study that include a more critical lens as to the study results. We ultimately conclude that real-world evidence (RWE) observations studies are here to stay.  They are needed along with more traditional randomized controlled trials (RCT).  A lively discussion, thanks in part to the work of StatNews, follows a review of the study.

Background: The FDA Contracts with The Study Authors to Form RCT Duplicate

The study was made possible by the U.S. Food and Drug Administration (FDA) contracting with both Brigham and Women’s Hospital, Harvard Medical School and health analytics venture Aetion. They entered into a program titled RCT Duplicate. The RCT Duplicate appears to be a new organization set up to implement real-world evidence-based trials.  The study authors involved in the CAROLINA study covered below are also included as key principals in the RCT Duplicate organization, including co-director Jessica Franklin and co-director RCT Duplicate vice chief Sebastian Schneeweiss who is both employed by  Brigham and Women’s Hospital, but also is a Co-Founder of Aetion.  We include a list of all of RCT Duplicate projects.

The Study 

The official title was “Predictions of Findings from the Ongoing CAROLINA Trial using Healthcare Database Analyses.

The study team sought to leverage real-world data (RWD) from three U.S. claims data sets. Their aim was to predict the findings of the CAROLINA trial comparing linagliptin vs. glimepiride in patients with type 2 diabetes (T2D) at increased cardiovascular risk by using a novel framework that requires passing pre-specified validity checks prior to analyzing the primary outcome.

A detailed description of the published RWD study ensues.  The CAROLINA is an going randomized, controlled trial (RCT) designed to assess whether linagliptin is non-inferior, and if so, superior, compared with glimepiride 1-4 mg once daily with respect to cardiovascular (CV) events in adults with relatively early Type 2 Diabetes at increased risk of CV events and with less than optimized glycemic control.

Given that medications of both classes are currently advocated as second-line therapy after metformin, and since sulfonylureas have been associated with concerns regarding their CV safety, while dipeptidyl peptidase-4 inhibitors have been suggested to exhibit CV benefits in preclinical and mechanistic trials, the results of this trial will provide answers to several clinically relevant questions and have a significant impact on clinical practice, according to the authors.

The authors declared this cohort study was initiated to predict the findings of CAROLINA trial in a real-world setting using electronic claims data from insurance database with results anticipated prior to the completion of CAROLINA.  Trial eligibility criteria were adopted in the claims data to generate a comparable study cohort (of linagliptin and glimepiride initiators) to that of the trial population. A 1:1 propensity scope-matching was used to control for >120 baseline characteristics.  Patients followed up with the study members for a composite cardiovascular outcome adapted from the primary end-point of the CAROLINA trial.

Study Design

An observational, retrospective, cohort type of study, the sponsors reviewed 50,000 participant records starting from May 1, 2011 and ending January 1, 2019.

Study Methods

The research team leveraged Medicare and two commercial claims data sets, and identified a 1:1 propensity-score-matched (PSM) cohort of T2D patients aged 40-85 at increased cardiovascular risk who initiated linagliptin or glimepiride. The team adapted eligibility criteria from CAROLINA to the data in the three different electronic health record data sets.

PSM was used to balance >120 confounders.  Validity checks included the evaluation of expected power, covariate balance and two control outcomes for which the team expected a positive association and a null finding.  They registered the RWD trial protocol (TrialSite News includes a link) prior to evaluating the composite cardiovascular outcome based on CAROLINA’s primary endpoint.

Hazard rations (HR) and 95% confidence intervals (CI) were eliminated in each data source and pooled with a fixed-effects meta-analysis.


The study team reported they identified 24,131 PSM pairs of linagliptin and glimepiride initiators with sufficient power for non-inferiority (>98%). Exposure groups achieved excellent covariate balance, including key lab results, and expected associations between glimepiride and hypoglycemia (HR=2.38 (95% CL=1.79-3.13)) and between linagliptin and end stage renal disease (HR=1.08 (0.66-1.79)) were replicated.  Linagliptin was associated with a 9% decreased risk in the composite cardiovascular outcome with a CI including the null (HR=0.91 (0.79-1.05)), in line with non-inferiority.

Study Team Conclusion

The RWD study team concluded that in a non-randomized, but rather observational, retrospective study leveraging electronic health record data (e.g. real-world data) that linagliptin has non-inferior risk of composite cardiovascular outcome compared to glimepiride.

Lead Research/Investigators

The authors for this study were associated with Brigham and Women’s Hospital and Harvard Medical School, as well as the Office of Medical Policy, Center for Drug Evaluation and Research, U.S. Food and Drug Administration.  However, one of the authors was associated with both the Brigham and Women’s Hospital and the RWD analytics company Aetion.

Elisabeth Patorno, MD, DrPH, Brigham and Women’s Hospital, Harvard

Sebastian Schneeweiss, MD, ScD, Harvard T.H. Chan, School of Public Health  and co-founder of Aetion

Chandrasekar Gopalakrishnan, MD, MPH, Brigham Women’s Hospital, Harvard

David Martin, MD, MPH, US FDA

Jessica Franklin, PhD, Brigham Women’s Hospital, Harvard

Research Lead Comments

Dr. Sebastian Schneeweiss is the lead author and co-founder of the vendor that supplied the data record analysis. He commented that the study was “quite an achievement” and “it has elevated the conversation from ‘real-world evidence is all bad’ to ‘let’s have a differentiated conversation about this,’ maybe there is something really good about this.”

Critical Commentary

StatNews published an article centering on the study. The author, Matthew Herper, was perhaps more critical than others would have liked.  After all, the combined FDA, Harvard and Aetion team is undertaking an unorthodox approach to generating and understanding clinical data. They are deviating from what, in a conservative-minded industry, has been a tried and true approach to clinical research: the Gold Standard interventional, random, controlled trial to FDA, GcP standards.

Herper reports although the FDA contracted with Aetion and Harvard for the RCT Duplicate initiative, the actual study covered herein comparing the CAROLINA trial is actually from a separate attempt by Aetion. Herper reports “which was being run by Boehringer Ingelheim and Eli Lilly to compare their diabetes drug, Tradjenta, to an older treatment, glimepiride. The pilot was funded by the FDA and Brigham (Harvard).”

Herper reports the study team did not publish their results but did present them at a scientific meeting of the American Diabetes Association on June 10.  The data reflected the actual results of the interventional study: that Tadjenta was in fact “non-inferior” in regard to reducing the combination of heart attacks, strokes and cardiovascular deaths, suggested Herper.

Some Differences

The StatNews observation suggested a not immaterial difference in the actual CAROLINA trial and the real-world trial simulation, that the actual reduction in hypoglycemia was bigger in the clinical trial than in the real-world trial simulation utilizing electronic health record data sets.

Ex-FDA Executive Supports the RWE Paradigm

Dr. Robert Califf has been a strong proponent of real-world evidence based research. Presently, he holds a position at Duke as professor and clinical investigator. It was announced a couple years ago that Califf would lead a new Duke University health data science-driven research and innovation collaborative center.

Dr. Califf said of the Aetion and Harvard-based RWE study that Aetion “does careful work” and the paper “looks solid,” reported StatNews.  Herper reported that Califf is a bog proponent of unorthodox thinking, leveraging technology and thinking a bit more out-of-the-box to get clinical trials productive again.  Dr. Califf understands the rapidly declining productivity clinical sponsors now face.  He has been a proactive and progressive voice in the industry advocating new approaches to research to drive economy and efficiency by leveraging electronic health records and RWE paradigm.

Critical View

Herper dug up some critics that offered differing perspectives, and they were not all supportive.  TrialSite News seeks to shine a light of transparency on clinical research with a site perspective—to benefit all—from patients and family members to physicians and care teams to investigator sponsors.

That being said, we reviewed StatNews, as well as other publications and spoke with several industry contacts that helps, we hope, share a balanced, candid-talk point of view.

StatNews Article

Herper spoke with a few physicians and researchers that offered up the following perspective.

Dr. Steven Nissen is the chief academic officer, Sydell, and Arnold Miller Family Heart and Vascular Institute at Cleveland Clinic. The Cleveland Clinic represents one of the top health institutions in the world, and Dr. Nissen has been with the prestigious provider since 1992.

He was quoted “I didn’t know whether to laugh or cry when I read this.”  He continued, “The fact that they got the right answer doesn’t mean its good research. And its not a good methodology and it’s not a substitute for careful and thoughtful prospective clinical trials.” Moreover, Herper elicited other critical comments from Nissen including “Is it useless? It’s not completely useless as a hypothesis-generating approach, but it is certainly not something that ought to be used for regulatory decisions.”  He adamantly declared “And it’s certainly not the type of study that should be used to make clinical decisions. Full stop.”  Nissan will never accept that RWE-based studies could supplant traditional interventional clinical trials. In fact, in a quote he included “how often have we been misled by observational research over the years?”

We here at TrialSite News in reading the study didn’t get the impression the RWE team was thinking of replacing traditional interventional, prospective clinical trials.  Rather, we see the FDA, Harvard and the analytics vendor looking at ways to better understand how RWE studies can augment the existing paradigm.

Dr. David Nathan, director of Diabetes Center and Clinical Research Center, Massachusetts General Hospital, a top diabetes expert was similarly critical in noting that it shouldn’t be thought of as a replacement or supplanting traditional clinical studies.  StatNews suggested that in Dr. Nathan’s world view, RWE studies should only be used for already understood bias correction.  Nathan noted to StatNews “We have to be really careful if you want to supplant what has really generated huge amounts of important data.”

When asked “how many times would Aetion have to replicate a clinical trial before Nathan believed the results?” Dr. Nathan responded “Infinite.”  Nathan suggests here from his response that RWE studies can never truly replace the gold standard clinical trial as we know it today.

Dr. Harlan Krumholz, director of the Center for Outcomes Research and Evaluation, Yale New Haven Hospital, noted concern about depending on claims data in such a way as Aetion does. However, he was more upbeat than the previous physicians, noting that he believed observational data could absolutely potential augment and support or even “fill the role of clinical trials in understanding diabetes drug safety,” reported StatNews.

Using RWE for Safety Signals

StatNews continued there could be a big opportunity to utilize the type of research included in the RCT Duplicate program to test the safety of diabetes drugs.  Herper introduced the Avandia example where the FDA mandated large clinical trials to determine safety.  It very well could be that a more efficient and economical approach would be to leverage RWE studies of the type the team conducted herein.  In the case of CAROLINA it took 8 years and lots of cost; in the case of the RWE model it could cut the time in half if not even more time—and of course cost.

Other Recent Literature Review

TrialSite News spoke with a number of experts and conducted a literature review to conclude that the future is bright for Real-World Evidence-based studies.

For example, recently in The Case for Real-World Evidence in the Future of Clinical Research on Chronic Myeloid Leukemia, Jonathan Webster and Douglas Smith of the Sidney Kimmel Comprehensive Cancer Center, Division of Hematologic Malignancies, Johns Hopkins University School of Medicine report that:

Real-world studies are different form conventional RCTs and therefore provide insight into distinct aspects of treatment and patient outcomes. Together with results from clinical trials, RWE can help to illustrate a more complete picture of the tolerability, effectiveness, and impact of a drug. The recently published guidelines indicate that the FDA expects a growing role of RWE.”

In Real-World Evidence Should be Used in Regulatory Decisions About New Pharmaceutical and Medical Device Products for Diabetes, a group of medical scholars suggest that there is a “Growing appreciation of the value of RWE to complement evidence from RTCs” and “there are growing number of opportunities for facilitating the regulatory approval processes for new drug and device therapies and diagnostics.”  The authors suggest some challenges in advancing the RWE approach involve “improving data collection and data quality,” not to mention “improving the means of analyzing relevant data to mitigate possible biases.”

In other studies, prominent groups of clinical investigators and prominent biopharmaceutical sponsors are commissioning studies to determine the extent to which RWE studies could support or augment random controlled trials such as Real-world Clinical Evidence on Rivaroxaban, Dabigatran, and Apixaban Compared with Vitamin K antagonists in Patients with Nonvalvular Atrial Fibrillation: A Systematic Literature Review where the study team found that “overall, the RWE studies were aligned with Phase 3 trials. However, conflicting results were reported for several outcomes of interest.”

Comparison of RWE and RCT

Characteristic RCTs RWE
Standard of Evidence Gold Standard Complementary to RCTs
Cost Costly to develop and conduct Less costly
Patient Population Well defined within constraints of specific inclusion criteria Broader and promotes evaluation of patient populations less often studies in RCTs
Sample Size Limited. Requires sample size calculation to be performed in advance Orders of magnitude larger
Efficacy Randomized and blinded lead to minimized risk of data bias and confounding Randomized and blinding may not be feasible; risk of unrecognized data and cofounding greater
Adverse Events Only more frequently occurring adverse events revealed Can reveal adverse events with much lower frequency and those requiring longer exposure to occur
Approval or Clearance of New Medical Products Considered the “Gold Standard” necessary for new drug approval Not generally accepted for approving new drugs; but can compliment RCT findings and accepted for new device indications
Role in Diabetes Define efficacy and provide a preliminary safety profile in a well-defined and controlled population Allows estimation of more realistic treatment effects of a wide range of diabetes interventions and evaluation interactions such as social determinants of health and comorbidities
Other Issues May be less useful when strong signals are available from RWE or early phase trials Facilitates post-marketing surveillance of adverse events and assessment of the product effectiveness; Results may be less credible due when a control group is not included

*table adopted from Gyawali et al 


Toward the end of 2018, the FDA launched a novel program, the Real-World Evidence Program, to promote the use of Real World Evidence (RWE) as part of its regulatory decision making processes for drugs and biologics.  The program was established by the 21st Century Cures Act.

As mentioned previously, the program’s designers thought to follow through on the intentions of the authors whose intention was to support the approval of new indications for already approved and marketed drugs or biologics or to help support post approval studies.

From the TrialSite News perspective, the RWE study train as left the station. Laws, regulators and a growing contingent in the medical and research community have been  bought in and fully believe in this paradigm as a way to support existing research methods and analysis.   RWE clearly has strengths and limitations and it can augment and support better and stronger in some cases rather than others.  The underlying assumptions, methods and factoring in of the totality of data and intelligence in a given research pursuit is of paramount importance for the highest quality results.

Source: Diabetes Journals

Pin It on Pinterest