Amund Vedal

In this study we have investigated how vision foundation models, pretrained on different domains, compete with a specialized model for classification as a function of the size of the labeled training set of medical images. Furthermore, we have looked into the different models' ability to generalize to difficult cases. Our experiments are conducted for cardiac ultrasound images and the downstream task of view recognition. Still, this classification task is meant to serve as a demonstrative example, where we think that the findings should be transferable to other classification tasks and other domains. Through these experiments we found that the foundation models were able to beat the performance of our task-specific supervised model when labelled training data were limited. This was true even for models trained on natural images and when using the simple linear probing method to create a classifier. We observed that more domain-specific foundation models achieved an even higher performance with limited data. On the other hand, the more general models showed a greater ability to generalize and perform well on difficult, out-of-distribution cases. Still, for typical in-domain cases with sufficient labeled data, a task-specific ResNet model was competitive with the foundation models, while also being both smaller and faster.

In this study we have investigated how vision foundation models, pretrained on different domains, compete with a specialized model for classification as a function of the size of the labeled training set of medical images. Furthermore, we have looked into the different models' ability to generalize to difficult cases. Our experiments are conducted for cardiac ultrasound images and the downstream task of view recognition. Still, this classification task is meant to serve as a demonstrative example, where we think that the findings should be transferable to other classification tasks and other domains. Through these experiments we found that the foundation models were able to beat the performance of our task-specific supervised model when labelled training data were limited. This was true even for models trained on natural images and when using the simple linear probing method to create a classifier. We observed that more domain-specific foundation models achieved an even higher performance with limited data. On the other hand, the more general models showed a greater ability to generalize and perform well on difficult, out-of-distribution cases. Still, for typical in-domain cases with sufficient labeled data, a task-specific ResNet model was competitive with the foundation models, while also being both smaller and faster.

Delineation of structures and estimation of landmarks in mammograms is a critical step in the evaluation of image quality in breast cancer screening, but requires the estimation of the uncertainty of the predicted landmarks to refer uncertain cases to clinicians. Of particular importance – and the focus of this work – is on the pectoral muscle, where the variability in muscle visibility across images introduces significant uncertainty. While graph convolutional networks (GCN) have been demonstrated to accurately predict landmarks by explicitly leveraging structural relationships between landmarks, they typically lack the ability to provide accurate uncertainty estimates for the landmarks. To address this shortcoming, in this work we propose a novel GCN-based approach that not only locates key points along the muscle boundary but also provides accurate uncertainty estimates, capturing both the aleatoric and epistemic uncertainties. Our method was evaluated on in-house annotated mammograms demonstrating comparable accuracy to human annotators, while at the same time providing highly accurate uncertainty estimates, confirming its potential for identifying cases that require human review. We further validate our proposed approach on the publicly available CSAW-S and INBreast datasets, demonstrating its robustness to domain shift, as well as its potential to detect incorrect or untypical annotations.

In this study, we apply the recent explainability methodologies Concept Relevance Propagation (CRP) and Relevance Maximization (RelMax), for understanding the decision-making processes of a deep learning classifier trained for mammogram-based breast cancer detection. The primary goal was to leverage CRP and RelMax to identify learned concepts that significantly influence the model's classifications. We aimed to present these in a manner intelligible to humans, with the intention of using this insight for quality assurance of the model. The study found that, although most high relevance concepts were shared across cancer subclasses, the model had formed a small number of concepts exclusive to our selected subclasses. These identified patterns seemed to capture well our intuitive understanding of what the given cancer subclass should look like. In addition, they also aligned well with the metadata information provided by radiologists and what they are typically familiar with. We believe this could further enhance their trust in the cancer classifier, since what the model has learned becomes more comprehensible and visually accessible. Our results refine a previously used explainability method, which was limited to localizing where the model directed its attention, but lacked in providing what was its understanding or offering additional clarification on the identified cancer subclass.