Value, Evidence and Outcomes

In February 2011 an IBM computer called Watson beat the two most successful champions of the quiz show Jeopardy!. Within months IBM had decided that healthcare would be Watson's first commercial market. Medical knowledge was growing faster than any doctor could read, and Watson, in IBM's framing, was a machine for reading huge amounts of text and ranking answers by evidence. In March 2012 IBM announced a collaboration with Memorial Sloan Kettering Cancer Center to train Watson in cancer treatment, and MD Anderson Cancer Center in Houston began its own project the same year. In 2015 IBM launched a Watson Health business unit and over the next year spent more than $4 billion on acquisitions, including Truven Health Analytics for about $2.6 billion.

The evidence never caught up with the marketing. Watson for Oncology was trained largely on a limited number of synthetic cases, hypothetical patients built by Sloan Kettering specialists, rather than on real outcomes. Published studies, mostly by its customers, measured how often it agreed with local tumour boards, and the results varied widely. A University of Texas System audit published in February 2017 found that MD Anderson had spent about $62.1 million on its project, which had never been used to treat a patient. In July 2018 STAT reported internal IBM slides citing "multiple examples of unsafe and incorrect treatment recommendations." By 2021 Watson Health had about $1 billion of revenue and no profit (reported), and in 2022 IBM sold the data and analytics assets to Francisco Partners for a price reported at about $1 billion.

Watson's failure was not that the software was bad at reading. It was that no one had asked the question every healthcare buyer eventually asks: compared with what, did patients do better? Agreement with doctors is not an outcome. If agreement is high, the tool adds little; if it is low, it raises a safety question. This chapter explains what counts as value in healthcare, what kind of evidence supports which claim, and why proof is the factor of the care equation that companies most often skip.