ai-for-mds-book.md (10093B)
1 # AI for MDs 2 3 ## Intro 4 5 ### Why? 6 7 - It's happening 8 - Data availability 9 - 10 11 --- 12 13 ### What is AI/ML? 14 15 16 AI 17 18 Tools for doing an intelligent thing (we're not going to get into AGI: while very cool, it's not super practical, and tends to obfuscate the useful stuff --> lead to debates on the nature of humanity, etc.) 19 20 Examples of simple AI: thermostats and cruise control. 21 A human who would like to maintain a constant speed would 22 A) pick the desired speed, 23 B) note whether they were currently above or below that speed, 24 and C) accelerate up or down to match the desired speed. 25 This is an example of a simple rules-based system, 26 a series of if-then statements (if above speed X, let off the gas. If below speed X, give it more gas). 27 These are very useful, and you can get a lot of mileage out of them, 28 but if decision-making is complex, then the series of interconnected rules become more complex, 29 and increased complexity leads to brittle systems with limited functionality and a high maintenance burden. 30 31 ML 32 33 Tools for training a machine to do a thing by giving it examples from which it can learn. We will get into the major categories later, but in brief they are 34 35 - Classification: learn from example whether a thing is an X or a Y. E.g. is this a picture of a hotdog or a shoe? Based on labs, genes, and demographics, does this patient have disease X or disease Y? 36 - Regression: learn from example the quantitative value of a thing based on other data. E.g. based on local property values, square footage, amenities, for how much will this house sell? Based on lab values that are easy to obtain, can we predict another lab value that is difficult to obtain (e.g. bone marrow cellularity, blast percentage). 37 - Unsupervised: from a set of data, find clusters of similar data points. E.g. from a number of whole genome sequences, find patterns of similarity that can be further explored and correlated with phenotype. In an image, locate areas likely to be of interest. 38 39 --- 40 41 ### Why you? 42 43 Data is king, queen, and jester 44 45 - [Breast cancer labels cheating](https://www.cs.princeton.edu/picasso/mats/KDDCup08Expl.pdf), [copy of PDF](http://beauhilton.com/assets/npj-dm-readmit.pdf) 46 47 The goal of this competition, organized by Siemens and available on the Kaggle platform 48 (which hosts various data science competitions that individuals or teams can freely compete in, often for a cash prize), 49 was to predict the likelihood of cancer from numeric variables derived from preprocessed images. 50 Anonymized patient identification numbers were left in the dataset, 51 and the winning competitors noticed that these numbers fell in certain ranges in a nonrandom way: 52 if a patient ID fell between 100,000 and 500,000, or above 4,000,000, the likelihood of cancer was very low (1-2%), 53 but if the ID was between 0 and 20,000 the likelihood of cancer was much higher (36%). 54 55 The competitors used this knowledge to build a model that performed very well in predicting which patients had cancer, 56 but would not generalize at all to the real world. 57 However, because the model was a "black box" model, 58 the problematic nature of their model would not be readily apparent, 59 and serves as a warning that machine learning practitioners 60 (and consumers of such technology) 61 should be wary of the data upon which models were built. 62 63 They wrote a lovely paper detailing their findings and warnings. 64 We can imagine much less scrupulous technologists letting their apparent success ride unchallenged. 65 66 - [Luke Oakden-Rayner predicting CT manufacturer from scans](https://www.nature.com/articles/s41746-019-0105-1), [copy of PDF](http://beauhilton.com/assets/predict-ct-scanner.pdf) 67 68 In a similar vein, Oakden-Rayner and colleagues used a deep learning model to predict various clinical entities from CT scans. 69 Particularly with deep learning models used for image analysis, 70 in which there may be millions or billions of parameters used to make the final prediction, 71 it can be very difficult to know exactly which features the model is pulling out of the image to make its predictions, 72 and in what ways those features interact. 73 The authors of this paper chose to turn things around a bit, 74 and give the model the task of predicting the manufacturer of the CT machine, 75 whether the order was stat or not, and other relatively non-clinical curiosities. 76 77 - [How to Ask an Actionable Question](https://www.thetalkingmachines.com/episodes/how-ask-actionable-question) 78 79 <relisten to this episode> 80 81 - [Missing weights](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7125114/), [copy of PDF](http://beauhilton.com/assets/npj-dm-readmit.pdf) 82 83 One of our projects sought to predict readmission risk and risk of increased length of stay, 84 from a number of hospital, demographic, laboratory, and other variables. 85 I chose to use a model that was amenable to explanation, 86 and could also handle missing values by allowing missingness itself to be a signal 87 (this is a confusing statement, but will be more clear with the following example). 88 In both predictive tasks, patients with extreme BMIs had higher predicted likelihood of the adverse outcome, 89 and, more unexpectedly, so did patients with *missing* BMIs. 90 91 I puzzled on this for some time, 92 wondering what a missing BMI could be signalling, 93 and then took care of a patient transferred from an outside hospital with a C1-C2 infection with high concern for instability. 94 He was never weighed out of concern that standing him up could precipitate a catastrophic event. 95 While this particular case is undoubtedly rare, 96 perhaps it is not so rare to have patients who are sick enough to never be stood up and put on a scale 97 (and I do not trust bed weights enough to include them in any predictive model). 98 99 I considered throwing this variable out several times because the results seemed, frankly, weird, 100 but time on the wards rescued it, 101 and now I find it emblematic of the communication between data science and clinical work. 102 Any hospital administrator looking to improve their prediction of which patients are high-risk 103 should consider looking for oddities such as this, 104 especially if the oddity is readily apparent in the hospital database, 105 and exploring anything that seems nonrandom, even if the connection is unclear at first. 106 107 - [Labelling heme-path images is hard](https://pubmed.ncbi.nlm.nih.gov/29656215/), [copy of PDF](http://beauhilton.com/assets/heme-path-is-hard.pdf) 108 109 I tried to build a deep learning model to predict hematologic malignancy phenotype from ultra-high-resolution scans of bone marrow biopsies. 110 I failed miserably. 111 112 It turns out this is a very hard problem. 113 Apart from considerable technical hurdles 114 (most deep learning models are trained on images 255x255 or 512x512 pixels, while these images are on the order of 80,000x90,000 pixels, requiring many decisions regarding pre- and post-processing to make the computation tractable), 115 we quickly realized that our data source was not rich enough and would need an enormous amount of work. 116 Many important findings are made on a cell-by-cell basis, 117 so we would need to label each cell to build a rich ground truth. 118 But labeling cells is no easy task, and requires experienced hematopathologists, 119 who are busy and expensive. 120 121 After some time I found I was not alone in realizing these difficulties, 122 and a group in Japan was way ahead of me and my team in tackling this problem. 123 This paper explicitly states their end goal, 124 to build an automated system to perform diagnoses on hematopathological images, 125 and then goes on to explain that they realized they needed more data, 126 and that collecting that data would require hematopathologists, 127 and that hematopathologists are busy and expensive, 128 so they would build a system to make the task of labelling easy and fast. 129 As a part of this, they wanted to evaluate the quality of the labels they were going to obtain, 130 and so performed a study to quantify agreement between hematopathologists for a select number of cellular entities and quantities. 131 They found consensus in some things, when they were obvious, 132 but nonconsensus in non-obvious things. 133 134 How, then, to build a model to automate part of a hematopathologist's job 135 (and hence to make advanced hematopathology available in less resource-rich settings), 136 if ground truth is so difficult to obtain, and shaky? 137 This basic problem is repeated throughout machine learning in medicine, 138 from improperly or incompletely labeled radiographs 139 to chart-based diagnoses based on billing concerns more than clinical realities. 140 We need experienced clinicians to generate, evaluate, and mobilize data, 141 and there is not (and, I argue, will never be) a substitute for direct experience with real patients 142 (or, in the cases of pathology and radiology, the tissues and images of real patients, and consultation with the clinicians who ordered the studies). 143 144 --- 145 146 ### How to learn 147 148 There is an ever-growing number of learning resources available for the would-be data scientist. 149 These range from in-person degrees at prestigious institutions, 150 to online courses and certifications, 151 to books, 152 blog posts, 153 and code examples that themselves range from the 154 line-by-line hands-on approach 155 to the highly cerebral and theoretic. 156 157 Which resources you choose to use will depend largely on your goals. 158 159 If you want to be a theoretician, 160 you may start from the ground up 161 and study the core mathematics and computer science. 162 163 If you want to be an engineer, 164 that is, one who applies the tools to real world problems, 165 you may prefer a hands-on approach at first, 166 and dip into the theory as needed to troubleshoot deeper issues. 167 168 You may want to be neither of these, 169 but instead your goal is to be a responsible citizen 170 within your medical field, 171 with enough understanding to critique the 172 academic articles and, 173 perhaps more importantly, 174 software tools employed within your practice and institution. 175 This may be analogous to the way and reasons we learn 176 statistics in medical school, 177 not to become statisticians, 178 but to know how to speak with statisticians, 179 how to collaborate, 180 how to understand what we read, 181 and spot poor practice 182 or misleading claims.