Will ChatGPT transform radiology?
Wednesday, 29 November, 2023
Dr Brendan Kelly (pictured) is HRB Wellcome Trust ICAT clinical academic fellow at St. Vincent's University Hospital Dublin, and the Insight Centre for Data Analytics at University College Dublin. A UCD graduate of radiography and medicine, his research interests are in the clinical application of artificial intelligence in radiology.
Listen to the (opens in a new window)podcast.
Dr Brendan Kelly said it was the first time he worried about whether artificial intelligence could take his job as a radiologist.
When the clinical academic fellow did an experiment to test ChatGPT4-Vision’s accuracy at predicting multiple sclerosis disease progression by assessing MRI scans, he was “so impressed” with the results.
Especially considering the latest iteration of OpenAI’s incredibly successful Large Language Models is just a month old.
Using 100 unseen and random paired images from 170 patients for testing, Dr Kelly compared GPT4V’s performance with trained medical imaging models U-Net and Vision Transformer (ViT).
“Overall ChatGPT4-Vision had a 85% accuracy of predicting progression in multiple sclerosis. It was a simplified version of the task because I didn't give it the most subtle or the most difficult cases,” he clarifies. “But it still performed remarkably well. The dedicated computer vision models were scoring around 94% accuracy and it was scoring 85% accuracy - and that’s without the pre-training.”
“The dedicated computer vision models were scoring around 94% accuracy and ChatGPT4-Vision was scoring 85% accuracy - and that’s without the pre-training.”
Another point in GPT4V’s favour, he says, is its usability, with a simple, no-code, drag and drop interface.
“To pre-train those models usually requires coding experience. So the fact that it’s so available and accessible is what makes it very interesting.”
He believes GPT4V has the potential to disrupt AI radiology research, but stresses that its misclassified cases, hallucinations and overly cautious non-answers confirm it is not ready for clinical use.
“Even though its performance wasn't as good as some current state of the art models, it still makes me think, if this is ChatGPT4, what about Chat5? What about Chat6?” he muses. “So it was an exciting series of experiments, I enjoyed doing it.”
His paper Spot the Difference: Can ChatGPT4-Vision Transform Radiology Artificial Intelligence? is available on medRxiv, the preprint server for Health Sciences, operated by Cold Spring Harbor Laboratory, a research and educational institution.
“It will go to peer review and we'll see what they make of it.”
Dr Kelly started the radiology training scheme in 2016, the same year as ‘godfather of AI’ Professor Geoffrey Hinton likened radiologists to Wile E Coyote, already over the cliff face and oblivious to the imminent fall.
“He said, definitively, that we should stop training radiologists in 2016 because it was clear that in five years' time there would be no need for them. In 2023 there's a higher demand for radiologists than ever."
“He said, definitively, that we should stop training radiologists in 2016 because it was clear that in five years’ time there would be no need for them. That had an impression on me when I was starting, but here I am in 2023, and there's a higher demand for radiologists than ever.”
Demand for medical imaging is rising at a faster pace than most other aspects of healthcare, according to the British Medical Journal, and Dr Kelly highlights the fact that “there aren’t enough MRI scanners in Ireland or radiographers to run the scanners we have”.
Instead of Prof. Hinton’s gloomy prediction, he believes that radiologists who use AI will replace radiologists who don’t.
“That quote is from Curtis Langlotz, Professor of Radiology at the centre for artificial intelligence in medicine and imaging in Stanford University, which is where I did my Fulbright scholarship. I studied under him for a little while and that is the opinion that I've adopted. That was in 2020 and even then things were less advanced.”
He also agrees with Eric Topol, another thought leader in the medical AI space, who says machines should ease doctors’ workload, allowing them more time with patients.
“He says artificial intelligence should enable this human connection and not take away from it,” says Dr Kelly, confirming that AI is already deployed in Irish hospitals.
“One of the places that it's probably most advanced is in CT stroke imaging to diagnose acute stroke. It's really important to diagnose stroke as quickly as possible and for that reason two different AI systems are installed in Beaumont Hospital that they are using right now to help triage those patients. The emergency department of the Mater hospital is really pioneering integrating many different AI algorithms into their practice. But all of them are what's being called augmented intelligence rather than artificial intelligence,” he differentiates. “These are decision support tools, rather than autonomous systems, that are making the job of the radiologist and the radiology trainees easier by trying to get to the most relevant scans as quickly as possible.”
St Vincent’s Hospital, meanwhile, is one of the centres of excellence for multiple sclerosis scanning in Ireland.
“Part of my research is around more timely and accurate diagnosis of MS, as well as trying to see if what's happening on the patient's current scan can predict what's going to happen in the future,” says Dr Kelly, who hopes AI will help radiologists devise more tailored treatment plans.
“So rather than getting a scan every six months, because that's what every patient gets, we would love to be able to move to a world where we can make a personalised decision.”
Some patients might need a scan every three months, he says, and others, every year, because of specific features in their MRI imaging.
“It would be lovely to be able to deliver personalised healthcare rather than a one-size-fits-all model.”
Listen to the (opens in a new window)podcast. More Rising Stars