Text Mining to Improve the Health of Millions of Citizens

By Kenji Takeda , Director of Azure for Research at Microsoft.

Doctors face daily decisions about the best care for their patients, and their own clinical experience can be enhanced using evidence-based medicine, such as through clinical trial data. As David Tovey, Editor-in-Chief, Cochrane, explained, "Before evidence-based medicine came along, people were reliant on the expertise of a doctor, the level of knowledge or understanding that he or she had. And this meant that treatments frequently took many, many years to come from research into practice."

One of the most robust ways of synthesizing research evidence across healthcare trials is through a systematic review. This involves finding, examining, and analyzing clinical trial data and research reports in a methodical way, to pull together high-quality summaries of how effective healthcare interventions are. This provides critical evidence to decision-makers at the international, national and local level, to make sure citizens receive the medical and social care they deserve. While this is a rigorous approach, it can take up to three years to produce a major systematic review, which limits our ability to use up to date research to guide decision-making.

Cochrane is a not-for-profit organization that creates, publishes and maintains systematic reviews of health care interventions, with more than 37,000 contributors working in 130 countries. The Cochrane Transform Project is using AI and machine learning to text mine thousands of reports to automatically select ones to include in systematic reviews. This saves weeks of monotonous work, freeing up the expert reviewers to spend their time and energy on high-level analysis. Researchers at University College London are using Azure Machine Learning to develop and deploy their text mining classifiers as a cloud service at scale, customized for different clinical assessment groups, in ways that were previously impossible. This is helping to make decisions around healthcare interventions faster and more accurate for millions of people around the world.


Teaching a Machine to Read Research Studies

The evidence pipeline developed by Cochrane is a 'surveillance' system that helps Cochrane find relevant research as soon as it is published. Research enters the pipeline through routine and specified searches of the health and social care literature and is then classified using machine learning. The three key types of classifier are grouped per:

  • Study type, e.g. is it a randomized control trial;
  • Review group, e.g. dementia, hypertension, pregnancy and childbirth, stroke, urology, etc.;
  • PICO characteristics:
    • Patient, population or problem, including characteristics such as demographics and risk factors;
    • Intervention under consideration for the patient or population;
    • Comparison with other interventions, e.g. placebo, or a different drug;
    • Outcome, e.g. quality of life, adverse effects, morbidity.

Cochrane evidence pipeline

The first stage in the pipeline is to identify research studies that are Randomized Control Trials (RCTs), so that we can filter out irrelevant studies quickly. To build these classifiers a training dataset was created using the Cochrane Crowd citizen science platform that enables anyone to contribute by helping to categorize medical research. A classifier was built using more than 300,000 records from Cochrane Crowd, including over 30,000 clinical trials. 60-80% of the studies have scores less than 0.1, so if we trust the machine, and automatically exclude these citations, we're left with 99.897% of the RCTs (i.e. we lose 0.1% but make significant gains in terms of manual workload reduction).

Randomized control trial classifier performance

Azure Machine Learning is used to provide text mining AI capabilities to speed up reviewing of clinical trial reports and research papers on healthcare interventions. The team easily moved their existing research methods in R to the cloud with Azure ML. A key advantage is that they can quickly create customized ML models for different end-users, e.g. groups looking at different clinical/medical conditions. "We've got a series of different classifiers which are running up on the Azure Machine Learning platform, where we prospectively, narrow the scope of what a particular citation is looking at. We have a study type classifier – the RCT classifier. I've just developed a systematic reviews classifier, which tells us about the type of study, the type of research we're looking at. Then we go into a domain-specific classifier. There are 50 odd Cochrane review groups, and we have a classifier for every review group. So we're able to say which review group, musculoskeletal group, heart group, all these different review groups, we can say which review group a particular citation is useful for." said Professor James Thomas, lead researcher at UCL.

The classification algorithms were originally written in R, and these could be seamlessly inserted into the Azure ML workflow. Over time, the researchers have moved to Python, and found it trivial to replace the R algorithms with Python ones in Azure ML. "Initially I developed these models in R, which didn't work as well as Python. So I was able to keep the top and the bottom of the process and just swap out the machine learning section in the middle.". Azure SQL Database is used to store the Cochrane database of trials and research studies, that is then used by Azure ML. These services are deployed through Azure ML APIs to help clinical assessment groups using the Cochrane Register of Studies (CRS) online service to select the studies that will be included in their reviews.

"What I like about building these classifiers using Azure Machine Learning is that I can deploy them as Web services at the click of a button and I don't have to maintain the virtual machine when I need to call them," Thomas added. The team is improving the PICO classifier, which is quite challenging, and using Azure N-Series GPU instances to speed up training of the neural network they are developing.

Cochrane Register of Studies (CRS)

Helping Millions of Patients and Citizens Receive the Best Healthcare

The same research group at UCL is also working with the National Institute for Health and Care Excellence (https://www.nice.org.uk/) in the U.K., which provides health care delivery guidance across the U.K. National Health Service (NHS) for more than 65 million people. The processes and technologies are similar – to use AI to build research 'surveillance' systems – and this time they will be used to update clinical guidelines. This work is deployed to hundreds of researchers worldwide through the EPPI Reviewer cloud service, which runs on Azure, using the APIs deployed through Azure ML. It will directly speed up adoption of new health care interventions at national and international scale, for example, by facilitating the clearing of new drugs for widespread use across the NHS. The result is that decisions about which interventions are best will be faster and more reliable, thanks to Microsoft's intelligent cloud.

Azure ML is augmenting systematic reviewers' capabilities, allowing them to focus on high-level analysis of research studies, instead of the monotonous work of sifting through thousands of research studies by hand. This new approach is leading to a system that is pro-active, classifying research as it is published, instead of relying on reviewers to do manual online searches before they can start analyzing the trial data. "What we're now doing is getting more granular and identifying which reviews a particular study might be relevant for. There are thousands of reviews in the Cochrane library, and what we're looking to do is be able to identify which review a new piece of research is relevant for, which will mean that the authors of that review can be alerted to its presence and potentially update the review if it looks as though the research would then change the review's findings," explained Thomas.

It is a critical advance, as currently Cochrane reviewers alone manually screen around three to four million citations every year. These developments at NICE and in the Cochrane Transform project are making major leaps to accelerate and improve the quality of healthcare decision-making globally.