index.html (17621B)
1 <!DOCTYPE html> 2 <html lang="en"> 3 <head> 4 <link rel="stylesheet" href="/style.css" type="text/css"> 5 <meta charset="utf-8"> 6 <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> 7 <meta name="viewport" content="width=device-width, initial-scale=1.0"> 8 <link rel="stylesheet" type="text/css" href="/style.css"> 9 <link rel="icon" href="data:image/svg+xml,<svg xmlns=%22http://www.w3.org/2000/svg%22 viewBox=%220 0 100 100%22><text y=%22.9em%22 font-size=%2290%22>🏕️</text></svg>"> 10 <title></title> 11 </head> 12 <body> 13 <div id="page-wrapper"> 14 <div id="header" role="banner"> 15 <header class="banner"> 16 <div id="banner-text"> 17 <span class="banner-title"><a href="/">beauhilton</a></span> 18 </div> 19 </header> 20 <nav> 21 <a href="/about">about</a> 22 <a href="/now">now</a> 23 <a href="/thanks">thanks</a> 24 <a class="nav-active" href="/posts">posts</a> 25 <a href="https://notes.beauhilton.com">notes</a> 26 <a href="https://talks.beauhilton.com">talks</a> 27 <a href="https://git.beauhilton.com">git</a> 28 <a href="/contact">contact</a> 29 <a href="/atom.xml">rss</a> 30 </nav> 31 </div> 32 <main> 33 <h1> 34 Writing clinical notes using AI - a practical experience 35 </h1> 36 <p> 37 <time id="post-date">2024-04-09</time> 38 </p> 39 <p id="post-excerpt"> 40 With Claude 3 Opus (a newish ChatGPT4 competitor) as my copilot, I wrote a pretty darn good SOAP note 41 for an (invented yet realistic) new consult visit. It took a little massaging to get it right, 42 but boy howdy, it did a great job. Here are the prompts and outputs. 43 This stuff has serious potential for streamlining the boring and tedious parts of the job. 44 </p> 45 <h2> 46 LLM, the Great and Terrible 47 </h2> 48 <p> 49 <a href="https://www.anthropic.com/claude">Claude</a> is one of 50 several new kids on the block in the land of highly capable Large 51 Language Models (LLMs), and is comparable to (but exceeding in certain 52 aspects) the incumbent ChatGPT4. 53 </p> 54 <p> 55 I decided to see how it would do writing a SOAP note for an imaginary 56 (but common and realistic) new patient consult in an oncology 57 clinic. 58 </p> 59 <p> 60 Why “the Great and Terrible?” Everybody knows by now that LLMs are 61 great at a lot of things, including writing coherent and fluent English 62 and conforming to pretty much any common text format you can find online 63 (from legal documents to Python or JavaScript code). Most folks also 64 know that LLMs aim to please, and will make up the details they lack in 65 order to satisfy your request, so their output cannot be trusted too 66 far, particularly in technical areas. My goal in this was to see how 67 close I could get to a usable clinic note, with full awareness that it 68 was going to hallucinate and require a tune-up on the back end. In other 69 words, I am confident LLMs are “terrible” in certain ways, but needed 70 convincing that they are “great” in this particular arena. 71 </p> 72 <h2> 73 Round 1: The Initial Prompt 74 </h2> 75 <p> 76 I asked Claude for the following: 77 </p> 78 <blockquote> 79 <p> 80 please write a maximally billable SOAP-style medical note for an 81 office visit with a new patient with likely stage III esophageal 82 squamous cell carcinoma. Note that we discussed the need for full 83 staging with FDG-PET CT to determine the final course of action, and the 84 risks and benefits of a CROSS-style chemoradiotherapy regimen if stage 85 III disease is confirmed vs systemic therapy if stage IV disease is 86 confirmed. The patient describes mild dysphagia to solid foods but 87 minimal weight loss, has no notable physical exam findings, and labs are 88 unremarkable other than mild anemia. I reviewed the available 89 cross-sectional imaging and concur with the radiologist’s report. 90 </p> 91 </blockquote> 92 <p> 93 And here’s the output: 94 </p> 95 <hr> 96 <p> 97 Here is a maximally billable SOAP-style medical note for the patient 98 encounter you described: 99 </p> 100 <p> 101 S (Subjective): 62 yo M with newly diagnosed esophageal cancer 102 presents to oncology clinic to establish care and discuss treatment 103 options. Reports mild dysphagia to solid foods x 3 weeks but denies 104 significant weight loss, odynophagia, chest pain, nausea/vomiting, 105 hematemesis, melena, hematochezia, or other GI symptoms. ROS negative 106 except as above. PMH notable for HTN, HLD, and GERD. FH neg for GI 107 malignancies. SH former smoker, quit 10 years ago, 30 pack-year history. 108 Married, retired accountant, lives at home with wife. Medications 109 include lisinopril, atorvastatin, omeprazole. No known allergies. 110 </p> 111 <p> 112 O (Objective):<br> 113 Vitals: Temp 98.6 F, BP 128/78, HR 72, RR 14, 98% RA, Wt 180 lb, Ht 70 114 in<br> 115 General: NAD, well-appearing. HEENT: PERRL, EOMI, oral mucosa moist. 116 Neck: no LAD. CV: RRR, no m/r/g. Pulm: CTAB. Abd: soft, NT/ND, no 117 masses. Ext: no c/c/e, 2+ pulses.<br> 118 Recent EGD: Esophageal mass 30 cm from incisors, partially obstructing 119 lumen, biopsies c/w squamous cell carcinoma. CT chest/abd/pelvis: 5 cm 120 distal esophageal mass, obliterating 75% of lumen. 2 cm regional lymph 121 node. No distant metastases identified. Radiologist assessment: T3N1M0, 122 clinical stage III. Labs: CBC: WBC 6.8, Hgb 11.5, Plt 225. CMP nl except 123 Alb 3.2. CEA 5.2. 124 </p> 125 <p> 126 A (Assessment): 127 </p> 128 <ul> 129 <li> 130 Esophageal squamous cell carcinoma, clinical stage III (T3N1M0). 131 Awaiting PET/CT for full staging and treatment planning. 132 </li> 133 <li> 134 Dysphagia secondary to above. Tolerating po currently without 135 significant loss. Monitor symptoms, intake closely. 136 </li> 137 <li> 138 H/o GERD on omeprazole. Continue current therapy. 139 </li> 140 <li> 141 Anemia, likely anemia of chronic disease but will evaluate as 142 outpatient. 143 </li> 144 </ul> 145 <p> 146 P (Plan): 147 </p> 148 <ol type="1"> 149 <li> 150 Will obtain FDG-PET/CT scan to complete staging. 151 </li> 152 <li> 153 If no distant metastatic disease, recommend neoadjuvant 154 chemoradiotherapy per CROSS regimen (carboplatin/paclitaxel q week x 5 + 155 41.4 Gy in 23 fractions). Discussed expected toxicities including 156 fatigue, esophagitis, nausea, neutropenia and importance of close 157 monitoring. 158 </li> 159 <li> 160 If metastatic disease identified, will pursue systemic chemotherapy, 161 likely FOLFOX or EOX regimen, rather than local therapy. May require 162 palliative measures for dysphagia (stent, dilation prn). 163 </li> 164 <li> 165 Check iron studies, B12, folate. Replace as indicated. 166 </li> 167 <li> 168 Nutrition consult. Encourage high calorie/protein foods. 169 </li> 170 <li> 171 RTC in 1 week to review PET/CT results and finalize treatment plan. 172 Sooner prn for worsening dysphagia, intolerance of po. 173 </li> 174 </ol> 175 <hr> 176 <p> 177 The LLM has added in a host of things I didn’t ask it for 178 specifically but do appreciate it including, which is what I hoped for 179 when I asked for a “maximally billable note” - e.g. the review of 180 systems, PMH, FH, SH, a reasonable assessment and plan for the anemia, 181 RTC, etc. Certain aspects are completely fabricated (age, vital signs, 182 the specifics of the radiology report, labs). Some things are 183 technically correct but not what I would do (e.g. very few people in the 184 US use the EOX regimen), so would need adjusting. I was a bit surprised 185 that it went ahead and put in the general plan for CROSS too - those 186 details are correct. 187 </p> 188 <h2> 189 Round 2: The Placeholder Marks 190 </h2> 191 <p> 192 I asked Claude to replace the invented values with placeholder marks 193 so I could fill them in later: 194 </p> 195 <blockquote> 196 <p> 197 That’s pretty good. Instead of inventing vital signs, imaging 198 reports, and lab values, please write it again to instead put in *** 199 placeholder marks in these areas so I can fill them in later. 200 </p> 201 </blockquote> 202 <p> 203 Claude obliged, creating a SOAP note template with *** all over the 204 place. It was a bit too much, as you can see: 205 </p> 206 <hr> 207 <p> 208 Sure, here’s the revised SOAP note with placeholder marks: 209 </p> 210 <p> 211 S (Subjective): *** yo M with newly diagnosed esophageal cancer 212 presents to oncology clinic to establish care and discuss treatment 213 options. Reports mild dysphagia to solid foods x *** weeks but denies 214 significant weight loss, odynophagia, chest pain, nausea/vomiting, 215 hematemesis, melena, hematochezia, or other GI symptoms. ROS negative 216 except as above. PMH notable for ***. FH *** for GI malignancies. SH *** 217 smoker, quit *** years ago, *** pack-year history. ***, ***, lives ***. 218 Medications include ***. *** allergies. 219 </p> 220 <p> 221 O (Objective): Vitals: Temp *** F, BP *<strong>/</strong>, HR ***, RR 222 ***, ***% RA, Wt *** lb, Ht *** in General: ***. HEENT: ***. Neck: ***. 223 CV: ***. Pulm: ***. Abd: <em><strong>. Ext: <em><strong>. Recent EGD: 224 Esophageal mass *** cm from incisors, partially obstructing lumen, 225 biopsies c/w squamous cell carcinoma. CT chest/abd/pelvis: *** cm distal 226 esophageal mass, obliterating <em><strong>% of lumen. *** cm regional 227 lymph node. No distant metastases identified. Radiologist assessment: 228 T</strong></em>N</strong></em>M</strong></em>, clinical stage ***. Labs: 229 CBC: WBC ***, Hgb ***, Plt ***. CMP *** except ***. CEA ***. 230 </p> 231 <p> 232 A (Assessment): 233 </p> 234 <ul> 235 <li> 236 Esophageal squamous cell carcinoma, clinical stage *** 237 (T<em><strong>N</strong></em>M***). Awaiting PET/CT for full staging and 238 treatment planning. 239 </li> 240 <li> 241 Dysphagia secondary to above. *** tolerating po currently without 242 significant *** loss. Monitor symptoms, intake closely. 243 </li> 244 <li> 245 *** on ***. Continue current therapy. 246 </li> 247 <li> 248 Anemia, likely anemia of chronic disease but will evaluate as 249 outpatient. 250 </li> 251 </ul> 252 <p> 253 P (Plan): 254 </p> 255 <ol type="1"> 256 <li> 257 Will obtain FDG-PET/CT scan to complete staging. 258 </li> 259 <li> 260 If no distant metastatic disease, recommend neoadjuvant 261 chemoradiotherapy per CROSS regimen (carboplatin/paclitaxel q week x 5 + 262 41.4 Gy in 23 fractions). Discussed expected toxicities including 263 fatigue, esophagitis, nausea, neutropenia and importance of close 264 monitoring. 265 </li> 266 <li> 267 If metastatic disease identified, will pursue systemic chemotherapy, 268 likely FOLFOX or EOX regimen, rather than local therapy. May require 269 palliative measures for dysphagia (stent, dilation prn). 270 </li> 271 <li> 272 Check iron studies, B12, folate. Replace as indicated. 273 </li> 274 <li> 275 Nutrition consult. Encourage high calorie/protein foods. 276 </li> 277 <li> 278 RTC in *** week(s) to review PET/CT results and finalize treatment 279 plan. Sooner prn for worsening dysphagia, intolerance of po. 280 </li> 281 </ol> 282 <hr> 283 <h2> 284 Round 3: The Goldilocks Zone 285 </h2> 286 <p> 287 Finally, I hit what seems to be a sweet spot: 288 </p> 289 <blockquote> 290 <p> 291 Forget what I said, please only put in *** placeholders for vital 292 signs and imaging. It’s ok to make up values for everything else. 293 </p> 294 </blockquote> 295 <p> 296 It still needs fixing, but isn’t so full of *** that it’s jarring (or 297 impossible to parse - what does “*** on ***” mean? Anything, and 298 nothing, that’s what.) 299 </p> 300 <hr> 301 <p> 302 Understood. Here’s the updated SOAP note with *** placeholders for 303 vital signs and imaging only: 304 </p> 305 <p> 306 S (Subjective): 62 yo M with newly diagnosed esophageal cancer 307 presents to oncology clinic to establish care and discuss treatment 308 options. Reports mild dysphagia to solid foods x 3 weeks but denies 309 significant weight loss, odynophagia, chest pain, nausea/vomiting, 310 hematemesis, melena, hematochezia, or other GI symptoms. ROS negative 311 except as above. PMH notable for HTN, HLD, and GERD. FH neg for GI 312 malignancies. SH former smoker, quit 10 years ago, 30 pack-year history. 313 Married, retired accountant, lives at home with wife. Medications 314 include lisinopril, atorvastatin, omeprazole. No known allergies. 315 </p> 316 <p> 317 O (Objective): Vitals: Temp *** F, BP *<strong>/</strong>, HR ***, RR 318 <em><strong>, <em><strong>% RA, Wt 180 lb, Ht 70 in General: NAD, 319 well-appearing. HEENT: PERRL, EOMI, oral mucosa moist. Neck: no LAD. CV: 320 RRR, no m/r/g. Pulm: CTAB. Abd: soft, NT/ND, no masses. Ext: no c/c/e, 321 2+ pulses. Recent EGD: Esophageal mass *** cm from incisors, partially 322 obstructing lumen, biopsies c/w squamous cell carcinoma. CT 323 chest/abd/pelvis: *** cm distal esophageal mass, obliterating 324 <em><strong>% of lumen. *** cm regional lymph node. No distant 325 metastases identified. Radiologist assessment: 326 T</strong></em>N</strong></em>M</strong></em>, clinical stage ***. Labs: 327 CBC: WBC 6.8, Hgb 11.5, Plt 225. CMP nl except Alb 3.2. CEA 5.2. 328 </p> 329 <p> 330 A (Assessment): 331 </p> 332 <ul> 333 <li> 334 Esophageal squamous cell carcinoma, clinical stage III (T3N1M0). 335 Awaiting PET/CT for full staging and treatment planning. 336 </li> 337 <li> 338 Dysphagia secondary to above. Tolerating po currently without 339 significant weight loss. Monitor symptoms, intake closely. 340 </li> 341 <li> 342 H/o GERD on omeprazole. Continue current therapy. 343 </li> 344 <li> 345 Anemia, likely anemia of chronic disease but will evaluate as 346 outpatient. 347 </li> 348 </ul> 349 <p> 350 P (Plan): 351 </p> 352 <ol type="1"> 353 <li> 354 Will obtain FDG-PET/CT scan to complete staging. 355 </li> 356 <li> 357 If no distant metastatic disease, recommend neoadjuvant 358 chemoradiotherapy per CROSS regimen (carboplatin/paclitaxel q week x 5 + 359 41.4 Gy in 23 fractions). Discussed expected toxicities including 360 fatigue, esophagitis, nausea, neutropenia and importance of close 361 monitoring. 362 </li> 363 <li> 364 If metastatic disease identified, will pursue systemic chemotherapy, 365 likely FOLFOX or EOX regimen, rather than local therapy. May require 366 palliative measures for dysphagia (stent, dilation prn). 367 </li> 368 <li> 369 Check iron studies, B12, folate. Replace as indicated. 370 </li> 371 <li> 372 Nutrition consult. Encourage high calorie/protein foods. 373 </li> 374 <li> 375 RTC in 1 week to review PET/CT results and finalize treatment plan. 376 Sooner prn for worsening dysphagia, intolerance of po. 377 </li> 378 </ol> 379 <hr> 380 <h2> 381 The Verdict 382 </h2> 383 <p> 384 Overall, writing the first LLM prompt, copying the result into the 385 chart, and adjusting it for correctness would take about 5 minutes. 386 Improving the prompt as I did adds some time, but you can keep whatever 387 prompt engineering bits you figure out for future use. You could also 388 build yourself a custom template and ask it to fill it out, which would 389 likely prevent many of the hallucinations (e.g. prepopulate placeholders 390 for the vitals, labs, imaging, and tell the LLM to leave those 391 alone). 392 </p> 393 <p> 394 Really, not bad for concise yet comprehensive documentation of a new 395 consult. For subsequent visits, copying the note forward and adjusting 396 for new details would be more efficient than going back to the LLM. 397 </p> 398 <p> 399 Here’s the crazy thing about all of the above: the LLM did not ingest 400 any PHI whatsoever - I gave it a general description of the case only. 401 No HIPAA is violated. No EMR integration is necessary - this would work 402 as well in CPRS as it would in Epic. Together, these two features are 403 powerful, and means you can use this starting today. 404 </p> 405 <p> 406 I know a lot of organizations are nervous about LLMs and AI in 407 medicine in general, but I’m not certain I understand the hesitation in 408 this particular arena. The responsibility is still yours for accurate 409 documentation. The plan is also still yours, developed together with the 410 patient. Many efficiencies can be gained without needing HIPAA-violating 411 specifics, and without somehow asking the AI to become the doctor. 412 </p> 413 <p> 414 Also, as a friend pointed out the other day, many, many clinical 415 notes are chock full of mistakes, whether typos or things copied forward 416 and not corrected, or more egregious things like true mistakes in 417 clinical reasoning. It seems to me, based on my brief experience, that 418 using a smarter Clippy would serve more to cut down these kinds of 419 errors, rather than propagating sins worse than those we already commit. 420 No electronic tool, whether a copy button in Epic or a powerful LLM, is 421 any kind of replacement for being a good clinician or an excuse for 422 shoddy work. But, on the same token*, I also don’t insist on counting 423 RBCs myself for every CBC I order - this is a good technological 424 cognate, since I need to be able to recognize the situations in which 425 the automated counter doesn’t tell the whole truth, know when the 426 results in the computer are fishy, and be able to do the visual 427 inspection myself if necessary, but most of the time I can just order 428 the CBC and go on my merry way. 429 </p> 430 <p> 431 * ha, “token.” There’s an LLM dad joke for ya. 432 </p> 433 <p> 434 P.S. I asked Claude to write a blog post summarizing my experience 435 and including the prompts and outputs. It mostly sucked and needed a 436 complete rewrite. But then again, I didn’t try very hard on the prompt. 437 I’ve tried asking it to write insurance request letters (e.g. for the 438 FDG-PET CT, which sometimes gets rejected out of hand even though it’s 439 2024), consult request letters, consult response letters, and patient 440 summary letters. Those were all pretty good, minimal tweaking 441 needed. 442 </p> 443 </main> 444 <div id="footnotes"></div> 445 <footer></footer> 446 </div> 447 </body> 448 </html>