Researchers conducted a narrative review and audit of vision-language foundation models (VLFMs) used for understanding medical images. They looked at 50 original studies and performed a detailed audit of 13 representative studies to see how well these AI models were being reported in scientific literature.
The audit found that many important metrics were missing from the reports. For example, only 31% of the audited studies reported human expert evaluations, and only 46% reported specific label evaluations. Other metrics, such as RadGraph F1 scores and inference speed, were not reported in any of the 13 audited studies.
Because these reports are often incomplete, it is difficult to know exactly how these models perform in real-world medical settings. The study notes that these models may also face issues like hallucination and limited interpretability. While these AI tools are promising, experts suggest that progress should only be judged by evidence that is traceable, clinically grounded, and validated by outside experts.
