Radiologists at Mass General Brigham began reading AI-flagged chest CT cases in 2019. By 2023, the institution had accumulated enough outcome data to run a retrospective analysis. The finding, published in Radiology, was not that AI made radiologists faster. It was that the specific radiologists who adopted AI tools showed no significant difference in miss rates compared to those who did not. The AI was being used. It was not changing outcomes.
This result is not a failure of the technology or the radiologists. It is the expected output of deploying phase-two AI into a phase-three world.
Phase One: Rule-Based CAD and the False Positive Problem
Computer-aided detection in radiology has a longer history than the deep learning narrative suggests. The first FDA-cleared CAD system for mammography launched in 1998. R2 Technology’s ImageChecker received clearance based on a dataset of 1,083 mammograms and a claimed 14% improvement in cancer detection. Hospitals bought it. Radiologists used it. And for a decade, CAD was the standard of care for screening mammography at any well-resourced institution.
The clinical evidence caught up in 2011. Lehman et al. published a retrospective analysis in JAMA covering 684,956 mammography examinations across 90 radiology facilities. Facilities that used CAD had a higher recall rate (meaning more women called back for additional imaging) but no statistically significant improvement in invasive cancer detection at 1 year. CAD was generating alerts. The alerts were not finding cancer. They were finding things that required follow-up, generating cost and patient anxiety, without producing a compensating benefit in outcomes.
The phase-one lesson: a system that is sensitive enough to flag suspicious regions is not necessarily specific enough to flag the right ones. Rule-based CAD was designed to maximize sensitivity on its training distribution. It learned to minimize missed cancers, not to minimize unnecessary recalls. These are different optimization targets, and the clinical system pays the cost of the latter.
Phase Two: The Deep Learning Benchmark Era
ImageNet changed the architecture of radiology AI research more than any clinical finding. Between 2016 and 2021, convolutional neural networks trained on large annotated medical imaging datasets produced a series of benchmark results that circulated as evidence that AI would replace radiologists within a decade.
The landmark papers are genuinely impressive. Rajpurkar et al. at Stanford published CheXNet in 2017, a 121-layer DenseNet trained on 112,120 chest X-rays that reported F1 scores matching or exceeding the average of four radiologists on pneumonia detection. McKinney et al. at Google reported in Nature in 2019 that a DL model for breast cancer screening outperformed an average of six radiologists, with a 5.7% reduction in false positives and a 9.4% reduction in false negatives on UK data. Ardila et al. reported in Nature Medicine that a DL model for lung cancer detection outperformed six radiologists on low-dose CT.
Every result was real on the test set it was measured against. Every comparison used radiologists reading images in isolation, without patient history, prior studies, or the ability to request additional imaging. Working radiologists do not interpret images this way. The benchmark measured a constrained version of radiologist performance against a model trained specifically on the task. The comparison was never between AI and clinical radiology. It was between AI and the subset of radiologist cognition that fits inside a forced-choice classification task.
Phase two also created a benchmark replication problem. External validation of published DL radiology models on independent test sets routinely showed performance drops of 10 to 20 percentage points. A 2021 Nature Machine Intelligence paper by Oakden-Rayner et al. found that multiple high-profile chest X-ray classification models had learned to predict from spurious features, including the radiographic markers attached to pacemakers, which are absent in healthy patients, rather than from pathology signals. The models were right for the wrong reasons.
Phase Three: Clinical Deployment Validation
The 556-paper bibliometric analysis by Li et al., published in European Radiology in 2025, is the most comprehensive systematic review of radiology AI clinical evidence available. The researchers categorized studies by whether they measured imaging performance metrics (benchmark accuracy) or patient-level clinical outcomes (what happened to the patient). Of studies reaching the clinical outcomes tier, the majority found no statistically significant improvement over standard of care.
Three mechanisms account for most of the gap. First, distribution shift: models trained on imaging data from academic medical centers fail on community hospital scanners using different reconstruction kernels, contrast protocols, and patient demographics. A model that achieves 94% AUC on Mass General Brigham data may achieve 78% on a rural Oklahoma hospital’s equipment. Second, alert fatigue: AI triage tools that flag 15% of cases as requiring urgent review generate radiologist fatigue when the positive predictive value is low. Radiologists learn to treat the AI output as noise. Third, workflow integration failure: AI outputs that arrive in the PACS system two hours after the radiologist has already read the case add zero clinical value regardless of accuracy.
The multicenter thymus CT validation study published in European Radiology in 2025 represents what phase-three success looks like. The task (segmenting thymic tissue from chest CT) has high inter-reader variability among human radiologists because the thymus involutes with age and its boundaries are genuinely ambiguous in adults. AI segmentation achieved lower inter-reader variability than the human baseline, on a clinically relevant task, across multiple institutions. The AI was solving a real problem where human performance was legitimately inconsistent. That is why the validation held.
Where AI Radiology Actually Works in 2026
Validated clinical gains cluster around applications with three shared properties: the task is well-defined, the consequence of delay is measurable, and the training distribution matches the deployment environment. Intracranial hemorrhage triage on non-contrast CT meets all three criteria. The finding is binary, the clinical consequence of a missed or delayed read is stroke progression, and the imaging protocol is standardized enough that distribution shift is manageable. Multiple prospective studies have confirmed that AI triage reduces door-to-treatment time in stroke workflows.
Lung nodule tracking software meets the first and third criteria. The Lung-RADS classification system provides a standardized framework, training data is abundant, and the benefit (reduced missed follow-up on incidental findings) is directly measurable in health system audit data. Automated bone age estimation from pediatric hand X-rays has shown reliable performance across demographics because the Greulich-Pyle atlas provides a standardized reference and the imaging protocol does not vary significantly between institutions.
Where AI radiology fails to demonstrate clinical benefit is in applications that require contextual clinical reasoning, integration of longitudinal patient data, or adaptation to imaging protocols not represented in the training distribution. These are not tractable with current architectures. Foundation models such as Merlin, covered separately in our analysis of the Nature Medicine paper, address the data efficiency problem by reducing the labeled examples required for fine-tuning. They do not resolve distribution shift on their own.
Limitations of the Current Evidence Base
No large randomized controlled trial has demonstrated that radiology AI integration reduces mortality, morbidity, or total cost of care at the population level. The clinical evidence base consists predominantly of retrospective single-center studies with known selection bias toward successful implementations. FDA clearance under the de novo pathway establishes safety and effectiveness relative to a predicate device, not superiority over standard of care. Cleared does not mean clinically beneficial.
Alert fatigue has not been rigorously measured across institutions. Most implementations do not publish their override rates, which are the most direct signal of whether radiologists trust the AI output. The models generating alerts are typically black boxes even under FDA clearance, and the interpretability tools available do not explain individual predictions in terms radiologists can validate against clinical reasoning.
The phase-three evidence gap is not a temporary problem that more deployment will resolve. It reflects a structural mismatch between how models are evaluated before deployment (controlled test sets, isolated image reading, single-institution data) and how they are used after deployment (real workflow integration, variable imaging protocols, longitudinal patient context). Closing that gap requires prospective study designs and outcome metrics that most deployments do not currently collect.
Primary sources: Li et al. 2025 bibliometric analysis, European Radiology. Lehman et al. 2011, JAMA. Rajpurkar et al. 2017 CheXNet, arXiv. McKinney et al. 2019, Nature. Ardila et al. 2019, Nature Medicine. Oakden-Rayner et al. 2021, Nature Machine Intelligence. FDA AI/ML medical device database.