TrialSite News was lucky enough to speak with Yizhao Ni, Ph.D., Assistant Professor, Department of Biomedical Informatics, Cincinnati Children’s Hospital Medical Center in Cincinnati, Ohio. Ni was part of a cross-functional team that developed an automated eligibility screening (ES) and trial-patient matching system for clinical trials in an urban tertiary care pediatric emergency department (ED); as well as evaluated the effectiveness of natural language processing (NLP) information extraction (IE) and machine learning (ML) techniques on real-world clinical trial data and trials. Yizhao Ni reported that the system produced material results in some scenarios reducing workload by 92% on the gold standard set, with a mean average precision (MAP) of 62.9%

Ni and team started the project back in 2012. The team sought to capitalize on emerging technologies, including machine learning and natural language processing as well as standardization of electronic health records (EHR) to streamline clinical trials patient enrollment results. Although clinical trial patient recruitment and enrollment have been a recognized, widely publicized challenge for decades now, biopharma sponsors and research centers continue to struggle with accelerating patient recruitment and enrollment initiatives.

Core Challenge

Many studies and reports have articulated the benefits of electronic health records for clinical trials-centered patient enrollment success. But in many cases, patient eligibility screening (ES) is still conducted in a manual and piecemeal way. Manual-based screening processes represent a burdensome and challenging effort—taking up great chunks of time. In a major center such as Cincinnati Children’s Hospital Medical Center, the effort of clinical trial patient screening less bias represents a significant labor-intensive process.

Industry Sponsors Patient Enrollment Challenges Magnify the Problem

The clinical trial phase is arguably the most expensive activity in a clinical trial program. Many industry sponsors have outsourced a great deal of clinical trial work to contract research organizations (CRO) in the hope of streamlining and driving efficiency. The results in some cases have gone well—in other cases not so well. With the advent of electronic health records those research sites that were large patient hubs saw new possibilities with the Affordable Care Act (ACA) in 2009. In the new law, providers were incentivized to purchase electronic health records (HER).

Electronic Health Records

By 2012 most major providers and research centers had implemented electronic health records (EHR). Often the larger centers implemented Cerner or Epic. Smaller federal qualified health centers (FQHCs) often selected eClincial Works. Across the nation researchers began to think about how to leverage the rich, robust data in these health record repositories: the identification of eligible participants automatically based on EHR information held great promise for translational science and clinical research.

Cincinnati Children Hospital Medical Center Makes the AI Move

By 2012, Ni and team sought to capitalize on technology advances and the rich data accumulating within the EHR—in their case Epic. It represented a real-world treasure trove of health informatics. The team embarked on what turned into a several year research project to transcend the “as is” situation at the time which involved manual eligibility screening to support clinical research. The team assembled developers, architects, information systems analysts and clinical trial subject matter experts not to mention clinical oncologists, to build a state-of-the-art machine learning-driven, natural language processing (NLP) and information extraction (IE) technology with the goal of improving the efficiency decision-making in clinical trial enrollment—with the “to be” objective of accelerating the ability to rapidly reduce the pool of potential patient participants for clinical coordinator and investigator staff screening.

The System: Data Collection and Organization for Machine Learning

The team spent extensive amounts of time in planning, execution, testing, refinement and ultimately the completion of this comprehensive, complex and ultimately valuable AI-based clinical trial-patient matching and enrollment system. The team had to collect Gold Standard data to train the machine learning algorithms for example. They collected narrative eligibility criteria from for 55 clinical trials actively enrolling oncology patients in Cincinnati Children’s Hospital Medical Center for example.

Simultaneously, the team developed an eligibility screening (ES) algorithm which they mobilized to extract clinical and demographic data from Epic, their EHR system data fields to showcase profiles of 215 oncology patients admitted for cancer treatment during this time for the test run.

System Execution: Real-Time Matching

Once the system was set up and testing several times over an extended period the automated ES algorithm was able to match clinical trial criteria with patient profiles greatly increasing the efficiency for potential trial-patient matches. The team validated the performance of the matches via a reference set of 169 historical trial-patient enrolment decision, workload, precision, recall, negative predictive value (NP) and specificity were calculated reported the Ni and other authors in their research article titled “Increasing the Efficiency of Trial-Patient Matching: Automated Clinical Trial Eligibility Pre Screening for Pediatric Oncology Patients.”


Ni and the team reported that if they didn’t have the automation option, a cancer investigator would need to review 163 patients per trial on average to reproduce historical patient enrollment for each study. Incredibly, the Cincinnati Children’s Hospital Medical Center system reduced the workload by 85% to 24 patients for this research endeavor. Put another way, with no AI-driven, natural language processing-based system, the clinical investigator would need to review 42 trials per patient on average to reproduce the patient-trial matches that occur in the retrospective data set. The net takeaway: Ni and team were able to automate 90% of the effort down to 4 trials saving enormous amounts of time and hence money.


The Cincinnati team designed a transformative system for clinical trials—think about the efficiency of any oncology practice once their trial-matching system can interface to any underlying EHR system. Ni reports that the solution can interface with any underlying EHR with open APIs. Their algorithm reduces the effort associated with patient recruitment and enrollment during a clinical trial. The volume and complexity of clinical trials only grows and intensifies so a tool like this if it could be pervasively deployed may make a noticeable impact.

Ni reports some lessons learned. He emphasized that the development of such a system isn’t necessarily a simple process. Often the biggest barrier to implement a system such as this is not the technology but rather the organization; the processes; the people. A big lesson for Ni that perhaps others can learn from is that to pull a project such as this off successfully requires a bridge between very different professional worlds of software development and architecture, information systems analysts, clinical trials operations and clinical trial subject matter experts not to mention therapeutic area domain expertise—e.g. oncology, etc. A unifying project driver must be in place.

When organizing for a project an individual(s) must be identified that represents the “bridge” to all others in the collaboration—the nexus or hub that drives the effort forward. They must be respected by the institution and the project stakeholders; they must understand the “languages” of technology, IT/IS, business analysts, and clinical subject matter expertise. Moreover, the individuals within the project nexus must be excellent communicators, coordinators and planners and frankly, possess a certain level of emotional intelligence to manage people in cross-functional, collaborative efforts.

It turns out that the most sophisticated of clinical trial systems—powered by natural language processing and machine learning—with demonstrable results are made successful by deeply committed, aligned and well-organized teams of experts with a central facilitator that ensures all team members move in a synchronized manner.

Contact for More Information

The Cincinnati Children’s Hospital Medical Center is a nonprofit institution dedicated to improving child health. The University of Cincinnati is a public research university. If other health systems and centers are conducting clinical research that is interested in learning more about the Cincinnati patient matching system they can contact Yizhao Ni, Assistant Professor, Department of Pediatrics, College of Medicine, the University of Cincinnati at email or telephone 513-803-4269.


Pin It on Pinterest