AI Case Study
MD Anderson Cancer Center project developing automated cancer treatment recommendation system with IBM Watson failed to deliver and was terminated
The University of Texas' MD Anderson Cancer Center wanted to develop its own IBM Watson for Oncology, a platform allowing natural language processing to provide treatment options for different types of cancers which had initially been developed with another hospital. But the partnership with IBM fell into cost and timeline overruns and was not successfully implemented.
Industry
Healthcare
Healthcare Providers And Services
Project Overview
Forbes reports: "MD Anderson would pay for the whole thing, eventually giving $39.2 million to IBM and $21.2 million to PricewaterhouseCoopers, which was hired to create a business plan around the product. Usually, companies pay research centers to do research on their products; in this case, MD Anderson paid for the privilege, although it would have apparently also owned the product. But the project disintegrated amid internal allegations of overspending, delays, and mismanagement... The cancer hospital’s first major challenge involved getting the machine to deal with the idiosyncrasies of medical records: the acronyms, human errors, shorthand phrases, and different styles of writing. 'Teaching a machine to read a record is a lot harder than anyone thought,' she said. Her team spent countless hours on that problem, trying to get Watson to extract valuable information from medical records so that it could apply them to its recommendations.
Finally, the project ran into a bigger obstacle: Even if you can get Watson to understand patient variables and make competent treatment recommendations, how do you get it access to enough patient data, from enough different sources, to derive insights that could significantly advance the standard of care? [Former project leader] Chin said that was a showstopper. Watson did not have a connected network of institutions feeding data about specific cohorts of patients. 'You may have 10,000 patients for lung cancer. That is still not a very big number when you think about it,' she said. With data from many more patients, Chin said, you could see patterns — 'subsets [of patients] that respond a certain way, subsets that don’t, subsets that have a certain toxicity. That pattern would help with better personalized and precision medicine. But we can’t get there without the ability to actually have a way of aggregating them'.”
Reported Results
The reason for the project termination as reported by Forbes is "the project was apparently seen as one that missed deadlines and didn’t deliver. The audit notes that the focus of the project was changed several times, first focusing on one type of leukemia, then another, then lung cancer. The initial plan was to test out the product out in pilots at two other hospitals. That never happened."
However, "IBM defended the MD Anderson product, known as the Oncology Expert Advisor or OEA. It says the OEA’s recommendations were accurate, agreeing with experts 90% of the time. 'The OEA R&D project was a success, and likely could have been deployed had MD Anderson chosen to take it forward,' says an IBM spokesperson."
Stat News claims "IBM told STAT that Chin’s work was separate from the effort to create Watson for Oncology, which was validated by cancer specialists at Memorial Sloan Kettering prior to its deployment. The company said that Watson for Oncology can extract and summarize substantial text from patient records, though the information must be verified by a clinician, and that it has made significant progress in obtaining more data to improve Watson’s performance."
From Forbes: "IBM now sells a product it developed with Memorial Sloan Kettering. The goal, as with the MD Anderson product, is to help doctors select treatments. Without a computer, this is done with a so-called “tumor board,” a group of experts who meet weekly. IBM points to a dozen studies presented at academic meetings showing that Watson’s recommendations agree with those of tumor boards. Several doctors who have examined Watson in other countries told STAT that Memorial Sloan Kettering’s role has given them pause. Researchers in Denmark and the Netherlands said hospitals in their countries have not signed on with Watson because it is too focused on the preferences of a few American doctors. Some hospitals abroad are customizing the system for their patients, adding information about local treatments. But he said doctors can find this localization redundant or unnecessary: They are not that interested in being told the same guidance they just taught Watson. Chen said this modified system is incredibly beneficial, however — to a hospital in the capital of Mongolia that employs zero oncology specialists."
Technology
Watson "provides supporting evidence for the recommendations it makes, but doesn’t actually explain how it came to recommend that particular treatment for that particular patient." (Stat News)
Function
Operations
General Operations
Background
From Forbes: "In March 2012, IBM signed a deal with Memorial Sloan Kettering Cancer Center in New York to develop a commercial product that would use the same technology to analyze the medical literature and help doctors choose treatments for cancer patients." Stat News reports that so far "more than 50 hospitals on five continents have agreements with IBM, or intermediary technology companies, to use Watson for Oncology to treat patients, and others are using the genomics and clinical trials products... MD Anderson Cancer Center at the University of Texas was among IBM’s first partners, and it was using the system to create its own expert oncology adviser, similar to the one IBM was developing with Memorial Sloan Kettering."
Benefits
Data
"The system is essentially Memorial Sloan Kettering in a portable box. Its treatment recommendations are based entirely on the training provided by doctors, who determine what information Watson needs to devise its guidance as well as what those recommendations should be. When users ask Watson for advice, the system also searches published literature — some of which is curated by Memorial Sloan Kettering — to provide relevant studies and background information to support its recommendation. But the recommendation itself is derived from the training provided by the hospital’s doctors, not the outside literature." (Stat News)