Michael C Kampffmeyer

Hyperbolic representation learning has shown compelling advantages over conventional Eu- clidean representation learning in modelling hierarchical relationships in data. In this work, we evaluate its potential to capture biological relations between cell types in highly multiplexed imaging data, where capturing subtle, hierarchical relationships between cell types is crucial to understand tissue composi- tion and functionality. Using a recent and thoroughly validated 42-marker Imaging Mass Cytometry (IMC) dataset of breast cancer tissue, we embed cells into both Euclidean and Lorentzian latent spaces via a fully hyperbolic variational autoencoder. We then introduce an information-theoretic framework based on k-nearest neighbour estimators to rigorously quantify the clustering performance in each geom- etry using mutual information and conditional mutual information. Our results reveal that hyperbolic embeddings retain significantly more biologically relevant information than their Euclidean counter- parts. We further provide open-source tools to extend Kraskov-Stögbauer-Grassberger based mutual information estimation to Lorentzian geodesic spaces, and to enable UMAP visualizations with hyper- bolic distance metrics. This work contributes a principled evaluation method for geometry-aware learning and supports the growing evidence of hyperbolic geometry’s benefits in spatial biology. Code is available at: https://github.com/youssefwally/FlatlandandBeyond

The Norwegian Centre for Research-based Innovation, Visual Intelligence, advances deep learning in marine science. The work focuses on analyzing multifrequency echosounder data to support ecosystem and fisheries management. To address limited labeled data for identifying sand eels in the North Sea, a semi-supervised learning method was developed that combines labeled and unlabeled data, significantly improving accuracy (Choi et al., ICES JMS 2021). Building on this, the method was extended to semantic segmentation (Choi et al., IEEE J. Ocean. Eng. 2023), enabling detailed classification of acoustic signals while reducing the need for costly annotations. More recently, foundation models were explored to tackle challenges like changing conditions in marine environments. By aligning these models with echosounder data and using semantic tokenization, they achieved strong performance with minimal labeled data (Choi et al., NAIS 2025). These innovations highlight the transformative role of AI in exploring and understanding the underwater world.

We demonstrate that hyperbolic representation learning effectively captures hierarchical cellular relationships in breast cancer. Using information-theoretic metrics, Lorentzian embeddings are shown to preserve significantly more biologically meaningful structure than Euclidean ones. Code: https://github.com/youssefwally/FlatlandandBeyond.

Tracer kinetic modeling serves a vital role in diagnosis, treatment planning, tracer development and oncology, but burdens practitioners with complex and invasive arterial input function estimation (AIF). We adopt a physics-informed CycleGAN showing promise in DCE-MRI quantification to dynamic PET quantification. Our experiments demonstrate sound AIF predictions and parameter maps closely resembling the reference.

Tracer kinetic modeling serves a vital role in diagnosis, treatment planning, tracer development and oncology, but burdens practitioners with complex and invasive arterial input function estimation (AIF). We adopt a physics-informed CycleGAN showing promise in DCE-MRI quantification to dynamic PET quantification. Our experiments demonstrate sound AIF predictions and parameter maps closely resembling the reference.

Hyperbolic representation learning has shown compelling advantages over conventional Euclidean representation learning in modeling hierarchical relationships in data. In this work, we evaluate its potential to capture biological relations between cell types in highly multiplexed imaging data, where capturing subtle, hierarchical relationships between cell types is crucial to understand tissue composition and functionality. Using a recent and thoroughly validated 42-marker Imaging Mass Cytometry (IMC) dataset of breast cancer tissue, we embed cells into both Euclidean and Lorentzian latent spaces via a fully hyperbolic variational autoencoder.

Hyperbolic representation learning has shown compelling advantages over conventional Euclidean representation learning in modeling hierarchical relationships in data. In this work, we evaluate its potential to capture biological relations between cell types in highly multiplexed imaging data, where capturing subtle, hierarchical relationships between cell types is crucial to understand tissue composition and functionality. Using a recent and thoroughly validated 42-marker Imaging Mass Cytometry (IMC) dataset of breast cancer tissue, we embed cells into both Euclidean and Lorentzian latent spaces via a fully hyperbolic variational autoencoder. We then introduce an information-theoretic framework based on k-nearest neighbor estimators to rigorously quantify the clustering performance in each geometry using mutual information and conditional mutual information. Our results reveal that hyperbolic embeddings retain significantly more biologically relevant information than their Euclidean counterparts. We further provide open-source tools to extend Kraskov-Stögbauer-Grassberger based mutual information estimation to Lorentzian geodesic spaces, and to enable UMAP visualizations with hyperbolic distance metrics. This work contributes a principled evaluation method for geometry-aware learning and supports the growing evidence of hyperbolic geometry's benefits in spatial biology. Code is available at: https://github.com/youssefwally/FlatlandandBeyond

Hyperbolic representation learning has recently emerged as a powerful framework for modeling hierarchical structures in data, often outperforming Euclidean embeddings. We investigate its utility for analyzing high-dimensional biological data from Imaging Mass Cytometry (IMC) of breast cancer tissues. We embed cells into Euclidean and Lorentzian latent spaces via a fully hyperbolic variational autoencoder (VAE) and propose an information-theoretic framework based on k-nearest neighbor estimators to quantify clustering quality using mutual information (MI) and conditional mutual information (CMI). Results show that Lorentzian embeddings preserve substantially more biologically relevant structure compared to Euclidean ones. We further provide open-source tools for Lorentzian MI estimation and hyperbolic UMAP visualization, enabling geometry-aware representation learning for spatial biology.

Clinical guidelines recommend performing left ventricular (LV) linear measurements in B-mode echocardiographic images at the basal level—typically at the mitral valve leaflet tips—and aligned perpendicular to the LV long axis along a virtual scanline (SL). However, most automated methods estimate landmarks directly from B-mode images for the measurement task, where even small shifts in predicted points along the LV walls can lead to significant measurement errors, reducing their clinical reliability. A recent semi-automatic method, EnLVAM, addresses this limitation by constraining landmark prediction to a clinician-defined SL and training on generated Anatomical Motion Mode (AMM) images to predict LV landmarks along the same. To enable full automation, a contour-aware SL placement approach is proposed in this work, in which the LV contour is estimated using a weakly supervised B-mode landmark detector. SL placement is then performed by inferring the LV long axis and the basal level—mimicking clinical guidelines. Building on this foundation, we introduce WiseLVAM—a novel framework for fully automated yet manually adaptable framework for automatically placing the SL and then automatically performing the LV linear measurements in the AMM mode. WiseLVAM utilizes the structure-awareness from B-mode images and the motion-awareness from AMM mode to enhance robustness and accuracy with the potential to provide a practical solution for the routine clinical application. The source code is publicly available at https://github.com/SFI-Visual-Intelligence/wiselvam.git.

The escalating demand for high-resolution Earth Observation (EO) data for various applications has significantly influenced advancements in image processing techniques. This study proposes a workflow to super-resolve the 12 spectral bands of Sentinel-2 Level-2A imagery to a ground sampling distance of 2.5m. The method leverages a hybrid approach, integrating advanced diffusion models with image fusion techniques. A critical component of the proposed methodology is super-resolution of Sentinel-2 RGB bands to generate a super-resolved Sentinel-2 RGB image, which subsequently serves in the image fusion pipeline that super-resolves the remaining spectral bands. The super-resolution algorithm is based on a diffusion model and is trained using the extensive National Agriculture Imagery Program (NAIP) dataset of aerial images, which is freely available. To make the super-resolution algorithm, trained on NAIP images, applicable to Sentinel-2 imagery, image harmonization and degradation were necessary to compensate for the inherent differences between NAIP and Sentinel-2 imagery. To address this challenge, we utilised a sophisticated degradation and harmonisation model that accurately simulates Sentinel-2 images from NAIP data, ensuring the harmonised NAIP images closely mimic the characteristics of Sentinel-2 observations post-resolution reduction. To investigate if learning the diffusion model using a large dataset of airborne images like NAIP provides better results than learning the model using a smaller satellite-based dataset like WorldStrat of high-resolution SPOT images, we performed a comparative analysis. The results demonstrate that models trained with the harmonised and correctly simulated datasets like NAIP significantly outperform those trained directly on SPOT images but also other existing super-resolution models available. This finding reveals that learning with more data can be beneficial if the data is properly harmonised and degraded to match the Sentinel-2 images. We performed a comprehensive evaluation using the recently established open-SR test methodology to validate the proposed model across multiple super-resolution metrics. This testing framework rigorously evaluates the super-resolution model based on metrics beyond traditional super-resolution metrics such as PSNR, SSIM, and LPIPS. Instead, the open-SR test evaluates the model based on metrics that measure its consistency, synthesis, and correctness. The proposed super-resolution model outperformed several current state-of-the-art models based on the comprehensive open-SR test framework. In addition, visual comparison further established the superior performance of our model in both urban and rural scenarios. An important component of the proposed model is the super-resolution of all 12 Sentinel-2 Level-2A bands, contrary to previous work, which has mainly focused on RGB band super-resolution. The proposed fusion pipeline successfully utilises the super-resolved image to obtain an enhanced 12-band Sentinel 2 image, similar to pansharpening techniques. We show qualitative and quantitative results on all 12 bands that demonstrate the seamless performance of the fusion method in super-resolution. This study not only showcases the potential of combining AI-driven super-resolution models with image fusion techniques in enhancing EO data resolution but also addresses the critical challenges posed by the diversity in data sources and the necessity for accurate generative models in training neural networks for super-resolution tasks.

Delineation of structures and estimation of landmarks in mammograms is a critical step in the evaluation of image quality in breast cancer screening, but requires the estimation of the uncertainty of the predicted landmarks to refer uncertain cases to clinicians. Of particular importance – and the focus of this work – is on the pectoral muscle, where the variability in muscle visibility across images introduces significant uncertainty. While graph convolutional networks (GCN) have been demonstrated to accurately predict landmarks by explicitly leveraging structural relationships between landmarks, they typically lack the ability to provide accurate uncertainty estimates for the landmarks. To address this shortcoming, in this work we propose a novel GCN-based approach that not only locates key points along the muscle boundary but also provides accurate uncertainty estimates, capturing both the aleatoric and epistemic uncertainties. Our method was evaluated on in-house annotated mammograms demonstrating comparable accuracy to human annotators, while at the same time providing highly accurate uncertainty estimates, confirming its potential for identifying cases that require human review. We further validate our proposed approach on the publicly available CSAW-S and INBreast datasets, demonstrating its robustness to domain shift, as well as its potential to detect incorrect or untypical annotations.

Foundation models (FMs) are transforming the field of Artificial Intelligence (AI) by learning inherent information from vast amounts of unlabeled data, enabling adaptation to numerous applications. Their integration into the Earth Observation (EO) ecosystem promises to revolutionize the information value chain, impacting industry, research, and science. However, EO applications present unique challenges, including the diverse needs for detail or rapid processing, and the variety of data value and sensor characteristics. Models must handle data from multiple sensors at varying ground sampling distances (GSD). Vision Transformers (ViT), often trained using self-supervised learning (SSL), form the backbone of many modern FMs by learning from complex data patterns without explicit supervision. We introduce FM4CS, a versatile foundation model specifically designed for climate and society EO applications. FM4CS aims to address the aforementioned challenges by supporting four different Sentinel sensors: Sentinel-1 SAR, Sentinel-2 MSI, Sentinel-3 OLCI, and Sentinel-3 SLSTR. Inspired by approaches like USat for multi-sensor data handling at native resolutions and FlexiViT for operating across a wide range of patch sizes without retraining, FM4CS employs a single ViT architecture. The model utilizes individual patch embedding layers for each sensor channel, allowing flexibility in processing subsets of spectral bands. It adapts the number of patches per band based on GSD and uses spectral group pooling to manage token sequence length. To accommodate flexible patch sizes, FM4CS incorporates a training procedure where patch size is randomized, allowing the model to adapt dynamically at inference time. For handling the need for positional information of the patches across different sensors, resolutions and patch sizes, FM4CS adapts the 2D ALiBi (Attention by Linear Bias) relative positional encoding scheme. The pre-training dataset for FM4CS is curated to ensure diversity. Instead of stacking small image crops, data is sampled using the Sentinel-2 tiling grid, co-locating Sentinel-1, Sentinel-2, and Sentinel-3 imagery for given locations and time intervals. A stratified sampling approach, based on k-means clustering of ESA WorldCover maps and Sentinel-2 RGB composites, is used to capture the diversity of global land cover and address imbalances. Oceanic data sampling incorporates shipping traffic density, oil and gas installations, and areas with higher probabilities of sea ice and icebergs. The dataset also includes ERA5-Land variables to facilitate multi-modal pretext tasks, leveraging daily statistics for variables such as soil moisture, temperature, and snow cover. FM4CS is trained using several SSL tasks. These include pixel-level input band reconstruction, similar to masked image modeling, where a lightweight ViT decoder reconstructs masked tokens. Additionally, the model predicts existing maps such as ESA WorldCover from Sentinel-1/2 data, and other land cover maps (ESA GlobCover, MOD12Q1) from the Sentinel-3 sensors using a cross-entropy loss. Image-level tasks involve the prediction of ERA5 variables, latitude, longitude and data acquisition month.

To leverage Earth observation (EO) data for large scale analysis, automatic methods is a prerequisite. Since 2012, deep learning (DL) models have brought about a revolutionary change in the analysis of image data and are currently considered state-of-the-art for a broad spectrum of EO tasks. However, a bottleneck with supervised DL models is that they often require a vast amount of labelled data to be trained, and the research community has therefore started to explore alternatives to supervised learning. During the last years, foundation models (FM) signify a change of thinking in computer vision. FMs are trained on a vast volume of unlabeled data and can identify complex patterns due to their large-scale learning capabilities. Typically, an additional head or decoder (small network) is added to the FM, which is trained and adapted to various use-cases by means of a small amount of labelled data. FM have also started to be explored for EO applications, however, current EO-based FMs are limited in terms of handling different modalities with large differences in resolution. Modern FMs are often based on transformers and are trained using self-supervised learning (SSL). There are several SSL schemes in place, including masked autoencoders (MAE) where we mask part of the input data and learn the model to predict the masked data. This is not useful by itself, but the model learns compressed representation of the data, which can be leveraged in downstream applications. This potentially makes the FM more useful than models trained on a limited set of labeled data. The Norwegian Computing Center and UiT – The Arctic University of Norway are in collaboration with user partners Romanian National Meteorological Administration, Danish Meteorological Institute, Polar View and Norwegian Water Resources and Energy Directorate developing a multi-modal FM. The FM is designed to process data from the satellites Sentinel-1 SAR, Sentinel-2 and Sentinel-3 OLCI and SLSTR. The FM is based on vison transformers (ViT) but utilizing the same principle as the USat approach to handle the different resolutions between the modalities. The training of the FM is based on the MAE approach, and to ensure that the SSL work efficiently, we have developed a smart sampling scheme during that provides relevant and diverse training data. In addition to SSL, we have also created a learning task in regressing climate variables from the ERA5 dataset. To train the FM, over 20 TB of Sentinel data was collected and processed using the LUMI supercomputer. The multi-modal FM is demonstrated on the following use-cases: mapping of snow, flood zone mapping, mapping and monitoring of sea ice, iceberg detection, early draught warning and mapping of wetlands. The resolution of the target use-case products are vastly different, e.g. for snow mapping we aim for a ground sampling distance (GSD) of 250m whereas for flood zone mapping we aim for a GSD of 10m. We have therefore trained two versions of the FM: one aiming for high-resolution products with GSD between 10 – 60m, and one aiming for low-resolutions products with GSD above 100m. The downstream tasks are implemented using the open-source framework TerraTorch, which is a flexible fine-tuning framework for geospatial FMs. TerraTorch supports common fine-tuning tasks such as image segmentation and pixel-wise regression along with a selection of task-specific decoder heads.

To leverage Earth observation (EO) data for large scale analysis, automatic methods is a prerequisite. Since 2012, deep learning (DL) models have brought about a revolutionary change in the analysis of image data and are currently considered state-of-the-art for a broad spectrum of EO tasks. However, a bottleneck with supervised DL models is that they often require a vast amount of labelled data to be trained, and the research community has therefore started to explore alternatives to supervised learning. Inspired by the progress in large language models, foundation models (FM) are now being applied extensively in computer vision. FMs are trained on a vast volume of unlabeled data and can identify complex patterns due to their large-scale learning capabilities. Typically, an additional head or decoder (small network) is added to the FM, which is trained and adapted to various use-cases by means of a small amount of labelled data. FM have also started to be explored for EO applications, however, current EO-based FMs are limited in terms of handling different modalities with large differences in resolution. Modern FMs are often based on transformers and are trained using self-supervised learning (SSL). There are several SSL schemes in place, including masked autoencoders (MAE) where we mask part of the input data and learn the model to predict the masked data. This is not useful by itself, but the model learns compressed representation of the data, which can be leveraged in downstream applications. This potentially makes the FM more useful than models trained on a limited set of labeled data. The Norwegian Computing Center and UiT – The Arctic University of Norway are in collaboration with user partners Romanian National Meteorological Administration, Danish Meteorological Institute, Polar View and Norwegian Water Resources and Energy Directorate developing a multi-modal FM. The FM is designed to process data from the satellites Sentinel-1 SAR, Sentinel-2 and Sentinel-3 OLCI and SLSTR. The FM is based on vison transformers (ViT) but utilizing the same principle as the USat approach to handle the different resolutions between the modalities. The training of the FM is based on the MAE approach, and to ensure that the SSL work efficiently, we have developed a smart sampling scheme during that provides relevant and diverse training data. In addition to SSL, we have also created a learning task in regressing climate variables from the ERA5 dataset. To train the FM, over 20 TB of Sentinel data was collected and processed using the LUMI supercomputer. The multi-modal FM is demonstrated on the following use-cases: mapping of snow, flood zone mapping, mapping and monitoring of sea ice, iceberg detection, early draught warning and mapping of wetlands. The downstream tasks are implemented using the open-source framework TerraTorch, which is a flexible fine-tuning framework for geospatial FMs. The FM4CS model is one of the first to apply Sentinel-3 data, which makes it attractive for climate applications.

Artificial intelligence tools are increasingly being used to analyse medical images. The team behind the MedEx project are developing new machine learning methods to glean information from medical images and help detect disease at an earlier stage than currently possible, as Professor Michael Kampffmeyer explains.

Common prototype-based medical image few-shot segmentation (FSS) methods model foreground and background classes using class-specific prototypes. However, given the high variability of the background, a more promising direction is to focus solely on foreground modeling, treating the background as an anomaly—an approach introduced by ADNet. Yet, ADNet faces three key limitations: dependence on a single prototype per class, a focus on binary classification, and fixed thresholds that fail to adapt to patient and organ variability. To address these shortcomings, we propose the Tied Prototype Model (TPM), a principled reformulation of ADNet with tied prototype locations for foreground and background distributions. Building on its probabilistic foundation, TPM naturally extends to multiple prototypes and multi-class segmentation while effectively separating non-typical background features. Notably, both extensions lead to improved segmentation accuracy. Finally, we leverage naturally occurring class priors to define an ideal target for adaptive thresholds, boosting segmentation performance. Taken together, TPM provides a fresh perspective on prototype-based FSS for medical image segmentation. The code can be found at https://github.com/hjk92g/TPM-FSS.

Regular mammography screening is essential for early breast cancer detection and deep learning-based risk prediction methods have sparked interest to adjust screening intervals for high-risk groups. While early methods focused only on current mammograms, recent approaches leverage the temporal aspect of screenings to track breast tissue changes over time, requiring spatial alignment across different time points. Two main strategies for this have emerged: explicit feature alignment through deformable registration and implicit learned alignment using techniques like transformers, with the former providing more control over the alignment. However, the optimal approach for explicit alignment in mammography remains underexplored. In this study, we provide insights into where explicit alignment should occur (input space vs. representation space) and if alignment and risk prediction should be jointly optimized. We demonstrate that jointly learning explicit alignment in representation space while optimizing risk estimation performance, as done in the current state-of-the-art approach, results in a trade-off between alignment quality and predictive performance and show that image-level alignment is superior to representation-level alignment, leading to better deformation field quality and enhanced risk prediction accuracy. The code is available at https://github.com/sot176/Longitudinal_Mammogram_Alignment.git.

Hyperbolic representation learning has shown compelling advantages over conventional Euclidean representation learning in modeling hierarchical relationships in data. In this work, we evaluate its potential to capture biological relations between cell types in highly multiplexed imaging data, where capturing subtle, hierarchical relationships between cell types is crucial to understand tissue composition and functionality. Using a recent and thoroughly validated 42-marker Imaging Mass Cytometry (IMC) dataset of breast cancer tissue, we embed cells into both Euclidean and Lorentzian latent spaces via a fully hyperbolic variational autoencoder. We then introduce an information-theoretic framework based on k-nearest neighbor estimators to rigorously quantify the clustering performance in each geometry using mutual information and conditional mutual information. Our results reveal that hyperbolic embeddings retain significantly more biologically relevant information than their Euclidean counterparts. We further provide open-source tools to extend Kraskov-Stögbauer-Grassberger based mutual information estimation to Lorentzian geodesic spaces, and to enable UMAP visualizations with hyperbolic distance metrics. This work contributes a principled evaluation method for geometry-aware learning and supports the growing evidence of hyperbolic geometry's benefits in spatial biology. Code is available at: https://github.com/youssefwally/FlatlandandBeyond

Incorporating uncertainty is crucial to provide trustworthy explanations of deep learning models. Recent works have demonstrated how uncertainty modeling can be particularly important in the unsupervised field of representation learning explainable artificial intelligence (R-XAI). Current R-XAI methods provide uncertainty by measuring variability in the importance score. However, they fail to provide meaningful estimates of whether a pixel is certainly important or not. In this work, we propose a new R-XAI method called REPEAT that addresses the key question of whether or not a pixel is certainly important. REPEAT leverages the stochasticity of current R-XAI methods to produce multiple estimates of importance, thus considering each pixel in an image as a Bernoulli random variable that is either important or unimportant. From these Bernoulli random variables we can directly estimate the importance of a pixel and its associated certainty, thus enabling users to determine certainty in pixel importance. Our extensive evaluation shows that REPEAT gives certainty estimates that are more intuitive, better at detecting out-of-distribution data, and more concise.

Hyperbolic representation learning has shown compelling advantages over conventional Euclidean representation learning in modeling hierarchical relationships in data. In this work, we evaluate its potential to capture biological relations between cell types in highly multiplexed imaging data, where capturing subtle, hierarchical relationships between cell types is crucial to understand tissue composition and functionality. Using a recent and thoroughly validated 42-marker Imaging Mass Cytometry (IMC) dataset of breast cancer tissue, we embed cells into both Euclidean and Lorentzian latent spaces via a fully hyperbolic variational autoencoder. We then introduce an information-theoretic framework based on k-nearest neighbor estimators to rigorously quantify the clustering performance in each geometry using mutual information and conditional mutual information. Our results reveal that hyperbolic embeddings retain significantly more biologically relevant information than their Euclidean counterparts. We further provide open-source tools to extend Kraskov-Stögbauer-Grassberger based mutual information estimation to Lorentzian geodesic spaces, and to enable UMAP visualizations with hyperbolic distance metrics. This work contributes a principled evaluation method for geometry-aware learning and supports the growing evidence of hyperbolic geometry's benefits in spatial biology. Code is available at: https://github.com/youssefwally/FlatlandandBeyond

Diffusion models have obtained photo-realistic results on various super-resolution tasks. However, existing approaches typically require the availability of high-resolution and paired training data, which often is not readily available in many remote sensing scenarios. To enhance multi-spectral Sentinel 2 (S2) satellite images – at a ground sampling distance (GSD) ranging from 10m to 60m – without requiring high-resolution or paired training data, we therefore propose and evaluate a novel set of approaches to leverage traditional pansharpening within a diffusion model context to simulate the required training data. We extensively compare the proposed methods and demonstrate that by utilizing unpaired Spot-6/7 data, we are able to produce photo-realistic S2 images at a resolution of 2.5m.

Group robustness strategies aim to mitigate learned biases in deep learning models that arise from spurious correlations present in their training datasets. However, most existing methods rely on the access to the label distribution of the groups, which is time-consuming and expensive to obtain. As a result, unsupervised group robustness strategies are sought. Based on the insight that a trained model's classification strategies can be inferred accurately based on explainability heatmaps, we introduce ExMap, an unsupervised two stage mechanism designed to enhance group robustness in traditional classifiers. ExMap utilizes a clustering module to infer pseudo-labels based on a model's explainability heatmaps, which are then used during training in lieu of actual labels. Our empirical studies validate the efficacy of ExMap - We demonstrate that it bridges the performance gap with its supervised counterparts and outperforms existing partially supervised and unsupervised methods. Additionally, ExMap can be seamlessly integrated with existing group robustness learning strategies. Finally, we demonstrate its potential in tackling the emerging issue of multiple shortcut mitigation.

Objective Function Mismatch (OFM) occurs when the optimization of one objective has a negative impact on the optimization of another objective. In this work we study OFM in deep clustering, and find that the popular autoencoder-based approach to deep clustering can lead to both reduced clustering performance, and a significant amount of OFM between the reconstruction and clustering objectives. To reduce the mismatch, while maintaining the structure-preserving property of an auxiliary objective, we propose a set of new auxiliary objectives for deep clustering, referred to as the Unsupervised Companion Objectives (UCOs). The UCOs rely on a kernel function to formulate a clustering objective on intermediate representations in the network. Generally, intermediate representations can include other dimensions, for instance spatial or temporal, in addition to the feature dimension. We therefore argue that the naïve approach of vectorizing and applying a vector kernel is suboptimal for such representations, as it ignores the information contained in the other dimensions. To address this drawback, we equip the UCOs with structure-exploiting tensor kernels, designed for tensors of arbitrary rank. The UCOs can thus be adapted to a broad class of network architectures. We also propose a novel, regression-based measure of OFM, allowing us to accurately quantify the amount of OFM observed during training. Our experiments show that the OFM between the UCOs and the main clustering objective is lower, compared to a similar autoencoder-based model. Further, we illustrate that the UCOs improve the clustering performance of the model, in contrast to the autoencoder-based approach. The code for our experiments is available at https://github.com/danieltrosten/tk-uco.

The recent development of self-explainable deep learning approaches has focused on integrating well-defined explainability principles into learning process, with the goal of achieving these principles through optimization. In this work, we propose DIB-X, a self-explainable deep learning approach for image data, which adheres to the principles of minimal, sufficient, and interactive explanations. The minimality and sufficiency principles are rooted from the trade-off relationship within the information bottleneck framework. Distinctly, DIB-X directly quantifies the minimality principle using the recently proposed matrix-based Rényi’s α-order entropy functional, circumventing the need for variational approximation and distributional assumption. The interactivity principle is realized by incorporating existing domain knowledge as prior explanations, fostering explanations that align with established domain understanding. Empirical results on MNIST and two marine environment monitoring datasets with different modalities reveal that our approach primarily provides improved explainability with the added advantage of enhanced classification performance.

Deep generative models with latent variables have been used lately to learn joint representations and generative processes from multi-modal data, which depict an object from different viewpoints. These two learning mechanisms can, however, conflict with each other and representations can fail to embed information on the data modalities. This research studies the realistic scenario in which all modalities and class labels are available for model training, e.g. images or handwriting, but where some modalities and labels required for downstream tasks are missing, e.g. text or annotations. We show, in this scenario, that the variational lower bound limits mutual information between joint representations and missing modalities. We, to counteract these problems, introduce a novel conditional multi-modal discriminative model that uses an informative prior distribution and optimizes a likelihood-free objective function that maximizes mutual information between joint representations and missing modalities. Extensive experimentation demonstrates the benefits of our proposed model, empirical results show that our model achieves state-of-the-art results in representative problems such as downstream classification, acoustic inversion, and image and annotation generation.

The analysis of maritime traffic patterns for safety and security purposes is increasing in importance and, hence, Vessel Traffic Service operators need efficient and contextualized tools for the detection of abnormal maritime behavior. Current models lack interpretability and contextualization of their predictions and are generally not quantitatively evaluated on a large annotated dataset comprising all expected traffic in a Region of Interest. We propose a model for the detection of abnormal maritime behaviors that provides the closest behaviors as context to the predictions. The normalcy model relies on two-step clustering, which is first computed based on the positions of the vessels and then refined based on their kinematics. We design for each step a similarity measure, which combined are able to distinguish boats cruising shipping lanes in different directions, but also vessels with more freedom, such as pilot boats. Our proposed abnormality detection model achieved, on a large annotated dataset extracted from AIS logs that we publish, an ROC-AUC of 0.79, which is on a par with State-of-the-Art deep neural networks, while being more computationally efficient and more interpretable, thanks to the contextualization offered by our two-step clustering.

Multi-frequency echosounder data can provide a broad understanding of the underwater environment in a non-invasive manner. The analysis of echosounder data is, hence, a topic of great importance for the marine ecosystem. Semantic segmentation, a deep learning based analysis method predicting the class attribute of each acoustic intensity, has recently been in the spotlight of the fisheries and aquatic industry since its result can be used to estimate the abundance of the marine organisms. However, a fundamental problem with current methods is the massive reliance on the availability of large amounts of annotated training data, which can only be acquired through expensive handcrafted annotation processes, making such approaches unrealistic in practice. As a solution to this challenge, we propose a novel approach, where we leverage a small amount of annotated data (supervised deep learning) and a large amount of readily available unannotated data (unsupervised learning), yielding a new data-efficient and accurate semi-supervised semantic segmentation method, all embodied into a single end-to-end trainable convolutional neural networks architecture. Our method is evaluated on representative data from a sandeel survey in the North Sea conducted by the Norwegian Institute of Marine Research. The rigorous experiments validate that our method achieves comparable results utilizing only 40 percent of the annotated data on which the supervised method is trained, by leveraging unannotated data. The code is available at https://github.com/SFI-Visual-Intelligence/PredKlus-semisup-segmentation.

Deep learning has the potential to revolutionize medical practice by automating and performing important tasks like detecting and delineating the size and locations of cancers in medical images. However, most deep learning models rely on augmentation techniques that treat medical images as natural images. For contrast-enhanced Computed Tomography (CT) images in particular, the signals producing the voxel intensities have physical meaning, which is lost during preprocessing and augmentation when treating such images as natural images. To address this, we propose a novel preprocessing and intensity augmentation scheme inspired by how radiologists leverage multiple viewing windows when evaluating CT images. Our proposed method, window shifting, randomly places the viewing windows around the region of interest during training. This approach improves liver lesion segmentation performance and robustness on images with poorly timed contrast agent. Our method outperforms classical intensity augmentations as well as the intensity augmentation pipeline of the popular nn-UNet on multiple datasets.

The fully supervised approaches achieve good performance provided that high-quality training data and an appropriate choice for the prediction model are assured. However, it is highly challenging for the echosounder data to obtain the class annotation for each backscattering intensity pixel because this relies on the manual annotation process, which is expensive and error-prone. As a key step in this direction, we propose a novel deep semi-supervised semantic segmentation method that efficient- ly uses a small amount of manually annotated data by com- bining it with a large amount of readily available unannotated data in the learning process. Poster presentation at VI days 2023. The original work is published in IEEE Journal of Oceanic Engineering DOI: 10.1109/JOE.2022.3226214.

Self-supervised learning is a central component in recent approaches to deep multi-view clustering (MVC). However, we find large variations in the development of self-supervision-based methods for deep MVC, potentially slowing the progress of the field. To address this, we present Deep-MVC, a unified framework for deep MVC that includes many recent methods as instances. We leverage our framework to make key observations about the effect of self-supervision, and in particular, drawbacks of aligning representations with contrastive learning. Further, we prove that contrastive alignment can negatively influence cluster separability, and that this effect becomes worse when the number of views increases. Motivated by our findings, we develop several new DeepMVC instances with new forms of self-supervision. We conduct extensive experiments and find that (i) in line with our theoretical findings, contrastive alignments decreases performance on datasets with many views; (ii) all methods benefit from some form of self-supervision; and (iii) our new instances outperform previous methods on several datasets. Based on our results, we suggest several promising directions for future research. To enhance the openness of the field, we provide an open-source implementation of DeepMVC, including recent models and our new instances. Our implementation includes a consistent evaluation protocol, facilitating fair and accurate evaluation of methods and components11Code: https://github.com/DanielTrosten/DeepMVC.

Distance-based classification is frequently used in transductive few-shot learning (FSL). However, due to the high-dimensionality of image representations, FSL classifiers are prone to suffer from the hubness problem, where a few points (hubs) occur frequently in multiple nearest neighbour lists of other points. Hubness negatively impacts distance-based classification when hubs from one class appear often among the nearest neighbors of points from another class, degrading the classifier's performance. To address the hubness problem in FSL, we first prove that hubness can be eliminated by distributing representations uniformly on the hypersphere. We then propose two new approaches to embed representations on the hypersphere, which we prove optimize a tradeoff between uniformity and local similarity preservation - reducing hubness while retaining class structure. Our experiments show that the proposed methods reduce hubness, and significantly improves transductive FSL accuracy for a wide range of classifiers 11Code available at https://github.com/uitml/noHub..

Deep learning-based approaches for content-based image retrieval (CBIR) of computed tomography (CT) liver images is an active field of research but suffer from some critical limitations. First,they are heavily reliant on labelled data, which can be challenging and costly to acquire. Second, they lack transparency and explainability, which limits the trustworthiness of deep CBIR systems. We address these limitations by: (1) Proposing a self-supervised learning framework that incorporates domain-knowledge into the training procedure, and (2) by providing the first representation learning explainability analysis in the context of CBIR of CT liver images. Results demonstrate improved performance compared to the standard self-supervised approach across several metrics, as well as improved generalization across datasets. Further, we conduct the first representation learning explainability analysis in the context of CBIR, which reveals new insights into the feature extraction process. Lastly, we perform a case study with cross-examination CBIR that demonstrates the usability of our proposed framework. We believe that our proposed framework could play a vital role in creating trustworthy deep CBIR systems that can successfully take advantage of unlabeled data.

Analyzing deep neural networks (DNNs) via information plane (IP) theory has gained tremendous attention recently to gain insight into, among others, DNNs’ generalization ability. However, it is by no means obvious how to estimate the mutual information (MI) between each hidden layer and the input/desired output to construct the IP. For instance, hidden layers with many neurons require MI estimators with robustness toward the high dimensionality associated with such layers. MI estimators should also be able to handle convolutional layers while at the same time being computationally tractable to scale to large networks. Existing IP methods have not been able to study truly deep convolutional neural networks (CNNs). We propose an IP analysis using the new matrix-based Rényi’s entropy coupled with tensor kernels, leveraging the power of kernel methods to represent properties of the probability distribution independently of the dimensionality of the data. Our results shed new light on previous studies concerning small-scale DNNs using a completely new approach. We provide a comprehensive IP analysis of large-scale CNNs, investigating the different training phases and providing new insights into the training dynamics of large-scale neural networks.

The lack of labeled data is a key challenge for learning useful representation from time series data. However, an unsupervised representation framework that is capable of producing high quality representations could be of great value. It is key to enabling transfer learning, which is especially beneficial for medical applications, where there is an abundance of data but labeling is costly and time consuming. We propose an unsupervised contrastive learning framework that is motivated from the perspective of label smoothing. The proposed approach uses a novel contrastive loss that naturally exploits a data augmentation scheme in which new samples are generated by mixing two data samples with a mixing component. The task in the proposed framework is to predict the mixing component, which is utilized as soft targets in the loss function. Experiments demonstrate the framework’s superior performance compared to other representation learning approaches on both univariate and multivariate time series and illustrate its benefits for transfer learning for clinical time series.

Innovation is considered essential for today's organizations to survive and thrive. Researchers have also stressed the importance of leadership as a driver of followers' innovative work behavior (FIB). Yet, despite a large amount of research, three areas remain understudied: (a) The relative importance of different forms of leadership for FIB; (b) the mechanisms through which leadership impacts FIB; and (c) the degree to which relationships between leadership and FIB are generalizable across cultures. To address these lacunae, we propose an integrated model connecting four types of positive leadership behaviors, two types of identification (as mediating variables), and FIB. We tested our model in a global data set comprising responses of N = 7,225 participants from 23 countries, grouped into nine cultural clusters. Our results indicate that perceived LMX quality was the strongest relative predictor of FIB. Furthermore, the relationships between both perceived LMX quality and identity leadership with FIB were mediated by social identification. The indirect effect of LMX on FIB via social identification was stable across clusters, whereas the indirect effects of the other forms of leadership on FIB via social identification were stronger in countries high versus low on collectivism. Power distance did not influence the relations.

Multi-modality data is becoming readily available in remote sensing (RS) and can provide complementary information about the Earth’s surface. Effective fusion of multi-modal information is thus important for various applications in RS, but also very challenging due to large domain differences, noise, and redundancies. There is a lack of effective and scalable fusion techniques for bridging multiple modality encoders and fully exploiting complementary information. To this end, we propose a new multi-modality network (MultiModNet) for land cover mapping of multi-modal remote sensing data based on a novel pyramid attention fusion (PAF) module and a gated fusion unit (GFU). The PAF module is designed to efficiently obtain rich fine-grained contextual representations from each modality with a built-in cross-level and cross-view attention fusion mechanism, and the GFU module utilizes a novel gating mechanism for early merging of features, thereby diminishing hidden redundancies and noise. This enables supplementary modalities to effectively extract the most valuable and complementary information for late feature fusion. Extensive experiments on two representative RS benchmark datasets demonstrate the effectiveness, robustness, and superiority of the MultiModNet for multi-modal land cover classification.

This paper presents the kernelized Taylor diagram, a graphical framework for visualizing similarities between data populations. The kernelized Taylor diagram builds on the widely used Taylor diagram, which is used to visualize similarities between populations. However, the Taylor diagram has several limitations such as not capturing non-linear relationships and sensitivity to outliers. To address such limitations, we propose the kernelized Taylor diagram. Our proposed kernelized Taylor diagram is capable of visualizing similarities between populations with minimal assumptions of the data distributions. The kernelized Taylor diagram relates the maximum mean discrepancy and the kernel mean embedding in a single diagram, a construction that, to the best of our knowledge, have not been devised prior to this work. We believe that the kernelized Taylor diagram can be a valuable tool in data visualization.

Acoustic target classification in multi-frequency echosounder data is a major interest for the marine ecosystem and fishery management since it can potentially estimate the abundance or biomass of the species. A key problem of current methods is the heavy dependence on the manual categorization of data samples. As a solution, we propose a novel semi-supervised deep learning method leveraging a few annotated data samples together with vast amounts of unannotated data samples, all in a single model. Specifically, two inter-connected objectives, namely, a clustering objective and a classification objective, optimize one shared convolutional neural network in an alternating manner. The clustering objective exploits the underlying structure of all data, both annotated and unannotated; the classification objective enforces a certain consistency to given classes using the few annotated data samples. We evaluate our classification method using echosounder data from the sandeel case study in the North Sea. In the semi-supervised setting with only a tenth of the training data annotated, our method achieves 67.6% accuracy, outperforming a conventional semi-supervised method by 7.0 percentage points. When applying the proposed method in a fully supervised setup, we achieve 74.7% accuracy, surpassing the standard supervised deep learning method by 4.7 percentage points.

Capturing global contextual representations in remote sensing images by exploiting long-range pixel-pixel dependencies has been shown to improve segmentation performance. However, how to do this efficiently is an open question as current approaches of utilising attention schemes, or very deep models to increase the field of view, increases complexity and memory consumption. Inspired by recent work on graph neural networks, we propose the Self-Constructing Graph (SCG) module that learns a long-range dependency graph directly from the image data and uses it to capture global contextual information efficiently to improve semantic segmentation. The SCG module provides a high degree of flexibility for constructing segmentation networks that seamlessly make use of the benefits of variants of graph neural networks (GNN) and convolutional neural networks (CNN). Our SCG-GCN model, a variant of SCG-Net built upon graph convolutional networks (GCN), performs semantic segmentation in an end-to-end manner with competitive performance on the publicly available ISPRS Potsdam and Vaihingen datasets, achieving a mean F1-scores of 92.0% and 89.8%, respectively. We conclude that the SCG-Net is an attractive architecture for semantic segmentation of remote sensing images since it achieves competitive performance with much fewer parameters and lower computational cost compared to related models based on convolutional neural networks.

In this essay, we review some recent advances in developing machine learning (ML) methods for marine science applications in Norway. We focus mostly on deep learning (DL) methods and review the challenges we have faced in the process, including data preparation, (lack of) labelled training data, and interpretability. We also present the partnerships that have been formed between e-science institutions and marine science institutions in Norway. These partnerships have been instrumental in moving this effort forward and have been fuelled by grants from the Norwegian Research Council. The last addition to this collaboration is the recent centres for research-based innovation in Marine Acoustic Abundance Estimation and Backscatter Classification (CRIMAC) and Visual Intelligence (VI).

Graph Neural Networks (GNNs) have received increasing attention in many fields. However, due to the lack of prior graphs, their use for semantic labeling has been limited. Here, we propose a novel architecture called the Self-Constructing Graph (SCG), which makes use of learnable latent variables to generate embeddings and to self-construct the underlying graphs directly from the input features without relying on manually built prior knowledge graphs. SCG can automatically obtain optimized non-local context graphs from complex-shaped objects in aerial imagery. We optimize SCG via an adaptive diagonal enhancement method and a variational lower bound that consists of a customized graph reconstruction term and a Kullback-Leibler divergence regularization term. We demonstrate the effectiveness and flexibility of the proposed SCG on the publicly available ISPRS Vaihingen dataset and our model SCG-Net achieves competitive results in terms of F1-score with much fewer parameters and at a lower computational cost compared to related pure-CNN based work.

We propose a novel architecture called the Multi-view Self-Constructing Graph Convolutional Networks (MSCG-Net) for semantic segmentation. Building on the recently proposed Self-Constructing Graph (SCG) module, which makes use of learnable latent variables to self-construct the underlying graphs directly from the input features without relying on manually built prior knowledge graphs, we leverage multiple views in order to explicitly exploit the rotational invariance in airborne images. We further develop an adaptive class weighting loss to address the class imbalance. We demonstrate the effectiveness and flexibility of the proposed method on the Agriculture-Vision challenge dataset and our model achieves very competitive results (0.547 mIoU) with much fewer parameters and at a lower computational cost compared to related pure-CNN based work.

Learning data representations that reflect the customers’ creditworthiness can improve marketing campaigns, customer relationship management, data and process management or the credit risk assessment in retail banks. In this research, we show that it is possible to steer data representations in the latent space of the Variational Autoencoder (VAE) using a semi-supervised learning framework and a specific grouping of the input data called Weight of Evidence (WoE). Our proposed method learns a latent representation of the data showing a well-defied clustering structure. The clustering structure captures the customers’ creditworthiness, which is unknown a priori and cannot be identified in the input space. The main advantages of our proposed method are that it captures the natural clustering of the data, suggests the number of clusters, captures the spatial coherence of customers’ creditworthiness, generates data representations of unseen customers and assign them to one of the existing clusters. Our empirical results, based on real data sets reflecting different market and economic conditions, show that none of the well-known data representation models in the benchmark analysis are able to obtain well-defined clustering structures like our proposed method. Further, we show how banks can use our proposed methodology to improve marketing campaigns and credit risk assessment.

Age-reading of fish otoliths (ear stones) is important for the sustainable management of fish resources. However, the procedure is challenging and requires experienced readers to carefully examine annual growth zones. In a recent study, convolutional neural networks (CNNs) have been demonstrated to perform reasonably well on automatically predicting fish age from otolith images. In the present study, we carefully investigate the prediction rule learned by such neural networks to provide insight into the features that identify certain fish age ranges. For this purpose, a recent technique for visualizing and analyzing the predictions of the neural networks was applied to different versions of the otolith images. The results indicate that supplementary knowledge about the internal structure improves the results for the youngest age groups, compared to using only the contour shape attribute of the otolith. However, the contour shape and size attributes are, in general, sufficient for older age groups. In addition, within specific age ranges we find that the network tends to focus on particular areas of the otoliths and that the most discriminating factors seem to be related to the central part and the outer edge of the otolith. Explaining age predictions from otolith images as done in this study will hopefully help build confidence in the potential of deep learning algorithms for automatic age prediction, as well as improve the quality of the age estimation.

Credit scoring models based on accepted applications may be biased and their consequences can have a statistical and economic impact. Reject inference is the process of attempting to infer the creditworthiness status of the rejected applications. Inspired by the promising results of semi-supervised deep generative models, this research develops two novel Bayesian models for reject inference in credit scoring combining Gaussian mixtures and auxiliary variables in a semi-supervised framework with generative models. To the best of our knowledge this is the first study coupling these concepts together. The goal is to improve the classification accuracy in credit scoring models by adding reject applications. Further, our proposed models infer the unknown creditworthiness of the rejected applications by exact enumeration of the two possible outcomes of the loan (default or non-default). The efficient stochastic gradient optimization technique used in deep generative models makes our models suitable for large data sets. Finally, the experiments in this research show that our proposed models perform better than classical and alternative machine learning models for reject inference in credit scoring, and that model performance increases with the amount of data used for model training.

Graph Neural Networks (GNNs) have received increasing attention in many fields. However, due to the lack of prior graphs, their use for semantic labeling has been limited. Here, we propose a novel architecture called the Self-Constructing Graph (SCG), which makes use of learnable latent variables to generate embeddings and to self-construct the underlying graphs directly from the input features without relying on manually built prior knowledge graphs. SCG can automatically obtain optimized non-local context graphs from complex-shaped objects in aerial imagery. We optimize SCG via an adaptive diagonal enhancement method and a variational lower bound that consists of a customized graph reconstruction term and a Kullback-Leibler divergence regularization term. We demonstrate the effectiveness and flexibility of the proposed SCG on the publicly available ISPRS Vaihingen dataset and our model SCG-Net achieves competitive results in terms of F1-score with much fewer parameters and at a lower computational cost compared to related pure-CNN based work.

The first Agriculture-Vision Challenge aims to encourage research in developing novel and effective algorithms for agricultural pattern recognition from aerial images, especially for the semantic segmentation task associated with our challenge dataset. Around 57 participating teams from various countries compete to achieve state-of-the-art in aerial agriculture semantic segmentation. The Agriculture-Vision Challenge Dataset was employed, which comprises of 21,061 aerial and multi-spectral farmland images. This paper provides a summary of notable methods and results in the challenge. Our submission server and leaderboard will continue to open for researchers that are interested in this challenge dataset and task; the link can be found here.

We propose a novel architecture called the Multi-view Self-Constructing Graph Convolutional Networks (MSCG-Net) for semantic segmentation. Building on the recently proposed Self-Constructing Graph (SCG) module, which makes use of learnable latent variables to self-construct the underlying graphs directly from the input features without relying on manually built prior knowledge graphs, we leverage multiple views in order to explicitly exploit the rotational invariance in airborne images. We further develop an adaptive class weighting loss to address the class imbalance. We demonstrate the effectiveness and flexibility of the proposed method on the Agriculture-Vision challenge dataset and our model achieves very competitive results (0.547 mIoU) with much fewer parameters and at a lower computational cost compared to related pure-CNN based work.

Land cover classification of remote sensing images is a challenging task due to limited amounts of annotated data, highly imbalanced classes, frequent incorrect pixel-level annotations, and an inherent complexity in the semantic segmentation task. In this article, we propose a novel architecture called the dense dilated convolutions' merging network (DDCM-Net) to address this task. The proposed DDCM-Net consists of dense dilated image convolutions merged with varying dilation rates. This effectively utilizes rich combinations of dilated convolutions that enlarge the network's receptive fields with fewer parameters and features compared with the state-of-the-art approaches in the remote sensing domain. Importantly, DDCM-Net obtains fused local- and global-context information, in effect incorporating surrounding discriminative capability for multiscale and complex-shaped objects with similar color and textures in very high-resolution aerial imagery. We demonstrate the effectiveness, robustness, and flexibility of the proposed DDCM-Net on the publicly available ISPRS Potsdam and Vaihingen data sets, as well as the DeepGlobe land cover data set. Our single model, trained on three-band Potsdam and Vaihingen data sets, achieves better accuracy in terms of both mean intersection over union (mIoU) and F1-score compared with other published models trained with more than three-band data. We further validate our model on the DeepGlobe data set, achieving state-of-the-art result 56.2% mIoU with much fewer parameters and at a lower computational cost compared with related recent work.

It is important, but challenging, for the forest industry to accurately map roads which are used for timber transport by trucks. In this work, we propose a Dense Dilated Convolutions Merging Network (DDCM-Net) to detect these roads in lidar images. The DDCM-Net can effectively recognize multi-scale and complex shaped roads with similar texture and colors, and also is shown to have superior performance over existing methods. To further improve its ability to accurately infer categories of roads,