AI Case Study

"We developed the AI Clinician, a computational model using reinforcement learning, which is able to dynamically suggest optimal treatments for adult patients with sepsis in the intensive care unit (ICU). Reinforcement learning is a category of AI tools in which a virtual agent learns from trial-and-error an optimized set of rules—a policy—that maximizes an expected return. Similarly, a clinician’s goal is to make therapeutic decisions in order to maximize a patient’s probability of a good outcome. Reinforcement learning has many desirable properties that may help medical decision-making. The intrinsic design of models using reinforcement learning can handle sparse reward signals, which makes them well-suited to overcome the complexity related to the heterogeneity of patient responses to medical interventions and the delayed indications of the efficacy of treatments11. Importantly, these algorithms are able to infer optimal decisions from suboptimal training examples. Reinforcement learning has been successfully applied in the past to medical problems, such as diabetes and mechanical ventilation in the ICU.

Our AI Clinician was built and validated on two large nonoverlapping ICU databases containing data routinely collected from adult patients in the United States. The Medical Information Mart for Intensive Care version III (MIMIC-III)18 was used for model development, and the eICU Research Institute Database (eRI) for model testing. In both datasets, we included adult patients fulfilling the international consensus sepsis-3 criteria. After exclusion of ineligible cases, we included 17,083 admissions (88.4% of eligible patients with sepsis) from five separate ICUs in one tertiary teaching hospital and 79,073 admissions (73.6% of eligible patients with sepsis) from 128 different hospitals from MIMIC-III and eRI, respectively (Supplementary Fig. 1).

In both datasets, we extracted a set of 48 variables, including demographics, Elixhauser premorbid status19, vital signs, laboratory values, fluids and vasopressors received (Supplementary Table 2). Patients’ data were coded as multidimensional discrete time series with 4-h time steps, and for each patient, we included up to 72 h of measurements taken around the estimated time of onset of sepsis. The total volume of intravenous fluids and maximum dose of vasopressors administered over each 4-h period defined the medical treatments of interest. The model aims at optimizing patient mortality, so a reward was associated to survival and a penalty to death.

A Markov decision process (MDP) was used to model the patient environment and trajectories20,21. The various elements of the model were defined using patient data time series from the training set (a random sample of 80% of MIMIC-III; Fig. 1). We deployed the AI Clinician to solve the MDP and predict outcomes of treatment strategies. First, we evaluated the actual treatments of clinicians by analyzing all the prescriptions and computing the average return of each treatment option, which can take values from –100 to + 100 in our model. Then, the MDP was solved using policy iteration, which identified the treatments that maximized return, that is, the expected 90-d survival of patients in the MIMIC-III cohort1.

We further validated the model by analyzing patient mortality when the dose actually administered corresponded to or differed from the dose suggested by the AI Clinician. Fifty-eight percent of the time, the patients received a dose of vasopressor very close to the suggested dose, within 0.02 µg/kg body weight/min (µg/kg/min) or 10% (whichever was smaller). For fluids, patients received the suggested dose approximately 36% of the time, within 10 mL/hour or 10%. These patients, who received doses similar to the doses recommended by the AI Clinician, had the lowest mortality. When the actual dose given was different from the suggested dose, clinicians gave more or less fluids in similar proportions and less vasopressor 75% of the time. Administering more or less of either treatment than the AI policy was associated with increasing mortality rates in a dose-dependent fashion. Fig. 3d,e depicts this association, with the dose gap averaged at the patient level. The median dose deficit in patients who received too little vasopressor was 0.13 µg/kg/min (interquartile range, 0.04–0.27 µg/kg/min).

Using a random forest classification model, we gained some insight into the model representations and interpretability by estimating the relative importance of the model parameters for predicting the administration of both medications (Supplementary Fig. 2). This confirmed that the decisions suggested by the AI Clinician were clinically interpretable and relied primarily on sensible clinical and biological parameters."

The research was conducted by scientists at the Department of Surgery and Cancer, the Department of Computer Science and of Bioengineering at Imperial College London, the Laboratory of Computational Physiology at Harvard–MIT Division of Health Sciences & Technology MIT, the Beth Israel Deaconess Medical Center in Boston, MA, the Department of eICU Research and Development, Philips Healthcare, Baltimore, MD, Department of Pharmacy Practice and Science, University of Maryland, School of Pharmacy, Baltimore, MD, Medical Research Council London Institute of Medical Sciences and the Behaviour Analytics Lab, Data Science Institute, London, UK.