Professor II
Robert Jenssen
- Department Image analysis, machine learning and Earth observation
- Phone number +47 41 69 96 12
- E-mail Robert.Jenssen@uit.no
Publications
- 381 publications found
Chen, Siyan; Wickstrøm, Kristoffer og Jenssen, Robert. (2026).
Evaluating AI-based Weather Forecasting Models for Local Wind Speed Prediction in Northern Norway. Robert Jenssen, Tian Tian, Tommy Sonne Alstrøm
Wu, Zhiyuan; Choi, Changkyu; Yu, Shujian; Jenssen, Robert og Ramezani-Kebrya, Ali. (2026).
Mitigating Embedding Leakage via Latent Disruption with Controlled Reconstruction.
Vis sammendrag
Pre-trained encoders produce semantically rich latent embeddings, which, however, may expose unintended information through malicious inference or exploitation. We propose SEAL, a framework that mitigates embedding leakage by disrupting latent representations based on information-theoretic principles. It reduces the risk of potential misuse while enabling controlled reconstruction for trusted users. SEAL learns to encode controlled perturbations by minimizing the Matrix Norm-based Quadratic Mutual Information (MQMI) functional between original and perturbed embeddings within a hyperspherical latent space. Meanwhile, a private decoder, jointly trained with the SEAL encoder, is trained to reconstruct the original data that is accessible only to authorized users under an access-controlled setting. Extensive experiments on vision and text datasets demonstrate that SEAL reduces latent leakage, weakens the effectiveness of evaluated inference attacks, and preserves reconstruction under the considered setting.
Østmo, Eirik Agnalt; Radiya, Keyur; Wickstrøm, Kristoffer; Kampffmeyer, Michael; Mikalsen, Karl Øyvind og Jenssen, Robert. (2026).
Liver, vessel, and tumor segmentation from partially labeled CT and multi-label masked learning.
Wetzer, Elisabeth; Handegard, Nils Olav; Kampffmeyer, Michael og Jenssen, Robert. (2026).
Problem-Driven AI Methodology for Fisheries Innovation. University of the Faroe Islands, Ministry of Foreign Affairs and Culture, Faroe Marine Research Institute
Vis sammendrag
The Norwegian Centre for Research-based Innovation, Visual Intelligence, advances deep learning in marine science. The work focuses on analyzing multifrequency echosounder data to support ecosystem and fisheries management. To address limited labeled data for identifying sand eels in the North Sea, a semi-supervised learning method was developed that combines labeled and unlabeled data, significantly improving accuracy (Choi et al., ICES JMS 2021). Building on this, the method was extended to semantic segmentation (Choi et al., IEEE J. Ocean. Eng. 2023), enabling detailed classification of acoustic signals while reducing the need for costly annotations. More recently, foundation models were explored to tackle challenges like changing conditions in marine environments. By aligning these models with echosounder data and using semantic tokenization, they achieved strong performance with minimal labeled data (Choi et al., NAIS 2025). These innovations highlight the transformative role of AI in exploring and understanding the underwater world.
Wetzer, Elisabeth; Choi, Changkyu; Jenssen, Robert; Handegard, Nils Olav og Ebbesson, Lars O.E.. (2026).
Artificial Intelligence for Sustainable Fisheries: Methods, Monitoring, and Practice. University of the Faroe Islands, Ministry of Foreign Affairs and Culture, Faroe Marine Research Institute
Vis sammendrag
Arctic and Subarctic fisheries face increasing pressure as traditional practices struggle to keep pace with shifting ocean conditions. These challenges call for intelligent, adaptive systems to support sustainable monitoring and management. This session explores how Artificial Intelligence (AI) can contribute to the future of fisheries science by bridging algorithmic innovation with practical application and fostering interdisciplinary exchange.
We invite papers that demonstrate how AI, ranging from machine learning and computer vision to emerging LLM-based agentic systems, can support fisheries research and regulation. Relevant applications include acoustic data interpretation for species identification, improved stock assessments, vessel activity monitoring, and adaptive regulatory strategies. Submissions addressing challenges such as limited computational resources, sparse data availability, or operational constraints in remote polar environments are particularly encouraged.
The session also highlights the importance of engaging Indigenous knowledge holders and coastal communities in AI development. We especially welcome co-designed frameworks that promote fairness, transparency, and local agency in contexts where environmental data and governance intersect.
Through this session, we aim to bring together diverse perspectives that advance responsible and locally grounded uses of AI for sustainable marine stewardship.
Jenssen, Robert; Eikvil, Line; Solberg, Anne H Schistad; Solheim, Inger og Bjørklund, Petter. (2026).
Visual Intelligence Annual Report 2025.
Salomonsen, Christian; Luppino, Luigi T.; Aspheim, Fredrik Emil; Wickstrøm, Kristoffer; Wetzer, Elisabeth; Kampffmeyer, Michael; Berzaghi, Rodrigo; Sundset, Rune; Jenssen, Robert og Kuttner, Samuel. (2026).
A robust and versatile deep learning model for prediction of the arterial input function in dynamic small animal [18F] FDG PET imaging.
Mylius-Kroken, Johan; Wetzer, Elisabeth; Ramezani-Kebrya, Ali; Jenssen, Robert og Wickstrøm, Kristoffer. (2025).
geobin: Geometric Binning Estimator. Integreat
NVA
poster
Møller, Bjørn Leth; Amiri, Sepideh; Igel, Christian; Wickstrøm, Kristoffer; Jenssen, Robert; Keicher, Matthias; Azampour, Mohammad Farid; Navab, Nassir og Ibragimov, Bulat. (2025).
NEMt: Fast Targeted Explanations for Medical Image Models via Neural Explanation Masks.
Vis sammendrag
A fundamental barrier to the adoption of AI systems in clinical practice is the insufficient transparency of AI decision-making. The field of Explainable Artificial Intelligence (XAI) seeks to provide human-interpretable explanations for a given AI model. The recently proposed Neural Explanation Mask (NEM) framework is the first XAI method to explain learned representations with high accuracy at real-time speed. NEM transforms a given differentiable model into a self-explaining system by augmenting it with a neural network-based explanation module. This module is trained in an unsupervised manner to output occlusion-based explanations for the original model. However, the current framework does not consider labels associated with the inputs. This makes it unsuitable for many important tasks in the medical domain that require explanations specific to particular output dimensions, such as pathology discovery, disease severity regression, and multi-label data classification. In this work, we address this issue by introducing a loss function for training explanation modules incorporating labels. It steers explanations toward target labels alongside an integrated smoothing operator, which reduces artifacts in the explanation masks. We validate the resulting Neural Explanation Masks with target labels (NEMt) framework on public databases of lung radiographs and skin images. The obtained results are superior to the state-of-the-art XAI methods in terms of explanation relevancy mass, complexity, and sparseness. Moreover, the explanation generation is several hundred times faster, allowing for real-time clinical applications. The code is publicly available at https://github.com/baerminator/NEM_T
Jenssen, Robert; Eikvil, Line; Solberg, Anne H Schistad; Solheim, Inger og Bjørklund, Petter. (2025).
Visual Intelligence Annual Report 2024.
Vis sammendrag
The Visual Intelligence Annual Report 2024 highlights the centre's progress, activities and achieved innovations for 2022. It describes new deep learning methods which address pressing societal needs in the fields of medicine and health, marine science, the energy sector, and earth observation.
Brüsch, Thea; Wickstrøm, Kristoffer; Schmidt, Mikkel N.; Alstrøm, Tommy Sonne og Jenssen, Robert. (2025).
FreqRISE: Explaining time series using frequency masking.
Vis sammendrag
Time series data is fundamentally important for many critical domains such as healthcare, finance, and climate, where explainable models are necessary for safe automated decision-making. To develop explainable artificial intelligence in these do mains therefore implies explaining salient information in the time series. Current methods for obtaining saliency maps assumes localized information in the raw input space. In this paper, we argue that the salient information of a number of time series is more likely to be localized in the frequency domain. We propose FreqRISE, which uses masking-based methods to produce explanations in the frequency and time-frequency domain, and outperforms strong baselines across a number of tasks. The source code is available here: https://github.com/theabrusch/FreqRISE.
Singh, Durgesh Kumar; Boubekki, Ahcene; Jenssen, Robert og Kampffmeyer, Michael. (2025).
SuperCM: Improving semi-supervised learning and domain adaptation through differentiable clustering.
Salomonsen, Christian; Kuttner, Samuel; Kampffmeyer, Michael; Jenssen, Robert; Wickstrøm, Kristoffer; Ye, Jong Chul og Wetzer, Elisabeth. (2025).
Fast Voxel-Wise Kinetic Modeling in Dynamic PET using a Physics-Informed CycleGAN. Medical Imaging Meets EurIPS (MedEurIPS)
Vis sammendrag
Tracer kinetic modeling serves a vital role in diagnosis, treatment planning, tracer development and oncology, but burdens practitioners with complex and invasive arterial input function estimation (AIF). We adopt a physics-informed CycleGAN showing promise in DCE-MRI quantification to dynamic PET quantification. Our experiments demonstrate sound AIF predictions and parameter maps closely resembling the reference.
Salomonsen, Christian; Kuttner, Samuel; Kampffmeyer, Michael; Jenssen, Robert; Wickstrøm, Kristoffer; Ye, Jong Chul og Wetzer, Elisabeth. (2025).
Fast Voxel-Wise Kinetic Modeling in Dynamic PET using a Physics-Informed CycleGAN.
Vis sammendrag
Tracer kinetic modeling serves a vital role in diagnosis, treatment planning, tracer development and oncology, but burdens practitioners with complex and invasive arterial input function estimation (AIF). We adopt a physics-informed CycleGAN showing promise in DCE-MRI quantification to dynamic PET quantification. Our experiments demonstrate sound AIF predictions and parameter maps closely resembling the reference.
Singh, Durgesh Kumar; Cao, Qing; Thomas, Sarina; Boubekki, Ahcène; Jenssen, Robert og Kampffmeyer, Michael. (2025).
WiseLVAM: A Novel Framework For Left Ventricle Automatic Measurements.
Vis sammendrag
Clinical guidelines recommend performing left ventricular (LV) linear measurements in B-mode echocardiographic images at the basal level—typically at the mitral valve leaflet tips—and aligned perpendicular to the LV long axis along a virtual scanline (SL). However, most automated methods estimate landmarks directly from B-mode images for the measurement task, where even small shifts in predicted points along the LV walls can lead to significant measurement errors, reducing their clinical reliability. A recent semi-automatic method, EnLVAM, addresses this limitation by constraining landmark prediction to a clinician-defined SL and training on generated Anatomical Motion Mode (AMM) images to predict LV landmarks along the same. To enable full automation, a contour-aware SL placement approach is proposed in this work, in which the LV contour is estimated using a weakly supervised B-mode landmark detector. SL placement is then performed by inferring the LV long axis and the basal level—mimicking clinical guidelines. Building on this foundation, we introduce WiseLVAM—a novel framework for fully automated yet manually adaptable framework for automatically placing the SL and then automatically performing the LV linear measurements in the AMM mode. WiseLVAM utilizes the structure-awareness from B-mode images and the motion-awareness from AMM mode to enhance robustness and accuracy with the potential to provide a practical solution for the routine clinical application. The source code is publicly available at https://github.com/SFI-Visual-Intelligence/wiselvam.git.
Thrun, Solveig; Hansen, Stine; Sun, Zijun; Blum, Nele; Salahuddin, Suaiba Amina; Wang, Xin; Wickstrøm, Kristoffer; Wetzer, Elisabeth; Jenssen, Robert; Stille, Maik og Kampffmeyer, Michael. (2025).
Reconsidering Spatial Alignment for Longitudinal Breast Cancer Risk Prediction. EurIPS Workshops
NVA
poster
Mylius-Kroken, Johan; Wickstrøm, Kristoffer; Wetzer, Elisabeth; Ramezani-Kebrya, Ali og Jenssen, Robert. (2025).
Can a Convex Partition caused by a CPWL Neural Network be used for Density Estimation?
Brüsch, Thea; Wickstrøm, Kristoffer; Schmidt, Mikkel N.; Jenssen, Robert og Alstrøm, Tommy Sonne. (2025).
FLEXtime: Filterbank Learning to Explain Time Series.
Vis sammendrag
State-of-the-art methods for explaining predictions from time series involve learning an instance-wise saliency mask for each time step; however, many types of time series are difficult to interpret in the time domain, due to the inherently complex nature of the data. Instead, we propose to view time series explainability as saliency maps over interpretable parts, leaning on established signal processing methodology on signal decomposition. Specifically, we propose a new method called FLEXtime that uses a bank of bandpass filters to split the time series into frequency bands. Then, we learn the combination of these bands that optimally explains the model’s prediction. Our extensive evaluation shows that, on average, FLEXtime outperforms state-of-the-art explainability methods across a range of datasets. FLEXtime fills an important gap in the current time series explainability methodology and is a valuable tool for a wide range of time series such as EEG and audio. Code is available at https://github.com/theabrusch/FLEXtime .
Dorszewski, Teresa; Tětková, Lenka; Jenssen, Robert; Hansen, Lars Kai og Wickstrøm, Kristoffer Knutsen. (2025).
From Colors to Classes: Emergence of Concepts in Vision Transformers.
Vis sammendrag
Vision Transformers (ViTs) are increasingly utilized in various computer vision tasks due to their powerful representation capabilities. However, it remains understudied how ViTs process information layer by layer. Numerous studies have shown that convolutional neural networks (CNNs) extract features of increasing complexity throughout their layers, which is crucial for tasks like domain adaptation and transfer learning. ViTs, lacking the same inductive biases as CNNs, can potentially learn global dependencies from the first layers due to their attention mechanisms. Given the increasing importance of ViTs in computer vision, there is a need to improve the layer-wise understanding of ViTs. In this work, we present a novel, layer-wise analysis of concepts encoded in state-of-the-art ViTs using neuron labeling. Our findings reveal that ViTs encode concepts with increasing complexity throughout the network. Early layers primarily encode basic features such as colors and textures, while later layers represent more specific classes, including objects and animals. As the complexity of encoded concepts increases, the number of concepts represented in each layer also rises, reflecting a more diverse and specific set of features. Additionally, different pretraining strategies influence the quantity and category of encoded concepts, with finetuning to specific downstream tasks generally reducing the number of encoded concepts and shifting the concepts to more relevant categories.
Uebbing, Lars; Joakimsen, Harald Lykke; Wickstrøm, Kristoffer; Kampffmeyer, Michael; Lefevre, Sebastien Francois; Salberg, Arnt Børre og Jenssen, Robert. (2025).
NOFE - Neural Operator Function Embedding. IRISA
NVA
poster
Uebbing, Lars; Joakimsen, Harald Lykke; Wickstrøm, Kristoffer; Kampffmeyer, Michael; Lefevre, Sebastien Francois; Salberg, Arnt Børre og Jenssen, Robert. (2025).
NOFE Neural Operator Function Embedding. IRISA
Mancisidor, Rogelio Andrade; Jenssen, Robert; Yu, Shujian og Kampffmeyer, Michael. (2025).
Aggregation of Dependent Expert Distributions in Multimodal Variational Autoencoders.
Thrun, Solveig; Hansen, Stine; Blum, Nele; Stille, Maik; Jenssen, Robert og Kampffmeyer, Michael. (2025).
TemporalMammoNet: Deep learning-based breast cancer classification using temporal mammograms. UiT Norges Arktiske Universitet
NVA
poster
Thrun, Solveig; Hansen, Stine; Sun, Zijun; Blum, Nele; Salahuddin, Suaiba Amina; Wickstrøm, Kristoffer; Wetzer, Elisabeth; Jenssen, Robert; Stille, Maik og Kampffmeyer, Michael. (2025).
Reconsidering Explicit Longitudinal Mammography Alignment for Enhanced Breast Cancer Risk Prediction. The Medical Image Computing and Computer Assisted Intervention Society
NVA
poster
Thrun, Solveig; Hansen, Stine; Sun, Zijun; Blum, Nele; Salahuddin, Suaiba Amina; Wickstrøm, Kristoffer; Wetzer, Elisabeth; Jenssen, Robert; Stille, Maik og Kampffmeyer, Michael. (2025).
Reconsidering Explicit Longitudinal Mammography Alignment for Enhanced Breast Cancer Risk Prediction.
Vis sammendrag
Regular mammography screening is essential for early breast cancer detection and deep learning-based risk prediction methods have sparked interest to adjust screening intervals for high-risk groups. While early methods focused only on current mammograms, recent approaches leverage the temporal aspect of screenings to track breast tissue changes over time, requiring spatial alignment across different time points. Two main strategies for this have emerged: explicit feature alignment through deformable registration and implicit learned alignment using techniques like transformers, with the former providing more control over the alignment. However, the optimal approach for explicit alignment in mammography remains underexplored. In this study, we provide insights into where explicit alignment should occur (input space vs. representation space) and if alignment and risk prediction should be jointly optimized. We demonstrate that jointly learning explicit alignment in representation space while optimizing risk estimation performance, as done in the current state-of-the-art approach, results in a trade-off between alignment quality and predictive performance and show that image-level alignment is superior to representation-level alignment, leading to better deformation field quality and enhanced risk prediction accuracy. The code is available at https://github.com/sot176/Longitudinal_Mammogram_Alignment.git.
Jenssen, Robert; Tegnander, Cathrine; Olsen, Dag Rune; Syrstad, Ragnhild Sjoner; Teigland, André; Thaulow, Sven Størmer; Bach, Kerstin og Fahlvik, Anne Kjersti. (2025).
Forskningsdrevet innovasjon innen KI: Hvordan styrker vi det sammen? SFI Visual Intelligence
Wickstrøm, Kristoffer; Brüsch, Thea; Kampffmeyer, Michael og Jenssen, Robert. (2025).
REPEAT: Improving Uncertainty Estimation in Representation Learning Explainability
| Proceedings of the AAAI Conference on Artificial Intelligence.
Vis sammendrag
Incorporating uncertainty is crucial to provide trustworthy explanations of deep learning models. Recent works have demonstrated how uncertainty modeling can be particularly important in the unsupervised field of representation learning explainable artificial intelligence (R-XAI). Current R-XAI methods provide uncertainty by measuring variability in the importance score. However, they fail to provide meaningful estimates of whether a pixel is certainly important or not. In this work, we propose a new R-XAI method called REPEAT that addresses the key question of whether or not a pixel is certainly important. REPEAT leverages the stochasticity of current R-XAI methods to produce multiple estimates of importance, thus considering each pixel in an image as a Bernoulli random variable that is either important or unimportant. From these Bernoulli random variables we can directly estimate the importance of a pixel and its associated certainty, thus enabling users to determine certainty in pixel importance. Our extensive evaluation shows that REPEAT gives certainty estimates that are more intuitive, better at detecting out-of-distribution data, and more concise.
Uebbing, Lars; Joakimsen, Harald Lykke; Luppino, Luigi Tommaso; Martinsen, Iver; McDonald, Andrew; Wickstrøm, Kristoffer; Lefevre, Sebastien Francois; Salberg, Arnt Børre; Hosking, Scott og Jenssen, Robert. (2025).
Investigating the Impact of Feature Reduction for Deep Learning-based Seasonal Sea Ice Forecasting.
Vis sammendrag
With the state-of-the-art IceNet model, deep learning has contributed to an important aspect of climate research by leveraging a range of climate inputs to provide accurate forecasts of Arctic sea ice concentration (SIC). The deep learning subfield of eXplainable AI (XAI) has gained enormous attention in order to gauge feature importance of neural networks, for instance by leveraging network gradients. In recent work, an XAI study of the IceNet was conducted, using gradient saliency maps to interrogate its feature importance. A majority of XAI studies provide information about feature importance as revealed by the XAI method, but rarely provide thorough analysis of effects from reducing the number of input variables. In this paper, we train versions of the IceNet with drastically reduced numbers of input features according to results of XAI and investigate the effects on the sea ice predictions, on average and with respect to specific events. Our results provide evidence that the model generally performs better when less features are used, but in case of anomalous events, a larger number of features is beneficial. We believe our thorough study of the IceNet in terms of feature importance revealed by XAI may give inspiration for other deep learning-based problem scenarios and application domains.
Choi, Changkyu; Subramaniam, Arangan; Handegard, Nils Olav; Ramezani-Kebrya, Ali og Jenssen, Robert. (2025).
Leveraging Foundation Model Adapters to Enable Robust and Semantic Underwater Exploration.
Vis sammendrag
This position paper presents a framework for intelligent underwater exploration by marrying foundation models (FMs) with multi‑frequency echosounder data. Echosounder data capture backscattered acoustic signals across a range of frequencies, providing rich insights into underwater environments by exploiting the frequency‑dependent scattering properties of underwater targets. However, their heterogeneity and complex structure complicate analysis. To address these challenges, the paper introduces four key innovations aimed at improving echosounder data analysis under dynamic ocean conditions: (1) aligning multi‑frequency echosounder data with FMs via lightweight FM adapters, (2) enabling continual adaptation to temporal distribution shifts in dynamic marine environments, (3) designing semantic tokenizers that preserve spatial structures, and (4) effectively leveraging sparse annotations to minimize dependence on costly labeled data. For each research direction, we map recent artificial intelligence (AI) methodologies to marine acoustic challenges and outline concrete pathways for technology transfer. Preliminary experiments demonstrate that a Vision Transformer (ViT), pretrained on natural images in a self-supervised manner, can segment sandeel schools from multi‑frequency echosounder data without task‑specific retraining. These results substantiate the proposed framework and illustrate the potential of cross‑disciplinary AI methods for ecologically informative underwater exploration.
Subramaniam, Arangan; Choi, Changkyu; Handegard, Nils Olav; Jenssen, Robert og Ramezani-Kebrya, Ali. (2025).
Marine Intelligence: Innovations to Enhance Underwater Exploration. NORA – Norwegian Artificial Intelligence Research Consortium
NVA
poster
Yu, Shujian; Li, Hongming; Løkse, Sigurd Eivindson; Jenssen, Robert og Principe, Jose C.. (2025).
The Conditional Cauchy-Schwarz Divergence With Applications to Time-Series Data and Sequential Decision Making.
Vis sammendrag
The Cauchy-Schwarz (CS) divergence was developed by Príncipe et al. in 2000. In this paper, we extend the classic CS divergence to quantify the closeness between two conditional distributions and show that the developed conditional CS divergence can be elegantly estimated by a kernel density estimator from given samples. We illustrate the advantages (e.g., rigorous faithfulness guarantee, lower computational complexity, higher statistical power, and much more flexibility in a wide range of applications) of our conditional CS divergence over previous proposals, such as the conditional Kullback-Leibler divergence and the conditional maximum mean discrepancy. We also demonstrate the compelling performance of conditional CS divergence in two machine learning tasks related to time series data and sequential inference, namely time series clustering and uncertainty-guided exploration for sequential decision making
Salahuddin, Suaiba Amina; Wetzer, Elisabeth; Wickstrøm, Kristoffer; Thrun, Solveig; Kampffmeyer, Michael og Jenssen, Robert. (2025).
Assessing the Efficacy of Multi-task Learning in Mammographic Density Classification: A Study on Class Imbalance and Model Performance.
Jenssen, Robert; Eikvil, Line; Solberg, Anne H Schistad; Solheim, Inger og Bjørklund, Petter. (2024).
Visual Intelligence Annual Report 2023.
Bjørklund, Petter; Kampffmeyer, Michael Christian; Salberg, Arnt-Børre og Jenssen, Robert. (2024).
Full klaff for KI-konferansen i Tromsø.
NVA
Populærvitenskapelig artikkel
Møller, Bjørn; Igel, Christian; Wickstrøm, Kristoffer Knutsen; Sporring, Jon; Jenssen, Robert og Ibragimov, Bulat. (2024).
Finding NEM-U: Explaining unsupervised representation learning through neural network generated explanation masks.
Vis sammendrag
Unsupervised representation learning has become an important ingredient of today’s deep learning systems. However, only a few methods exist that explain a learned vector embedding in the sense of providing information about which parts of an input are the most important for its representation. These methods generate the explanation for a given input after the model has been evaluated and tend to produce either inaccurate explanations or are slow, which limits their practical use. To address these limitations, we introduce the Neural Explanation Masks (NEM) framework, which turns a fixed representation model into a self-explaining model by augmenting it with a masking network. This network provides occlusion-based explanations in parallel to computing the representations during inference. We present an instance of this framework, the NEM-U (NEM using U-net structure) architecture, which leverages similarities between segmentation and occlusion-based masks. Our experiments show that NEM-U generates explanations faster and with lower complexity compared to the current state-of-the-art while maintaining high accuracy as measured by locality.
Jenssen, Robert. (2024).
MAP IT to Visualize Representations.
Vis sammendrag
MAP IT visualizes representations by taking a fundamentally different approach to dimensionality reduction. MAP IT aligns distributions over discrete marginal probabilities in the input space versus the target space, thus capturing information in wider local regions, as opposed to current methods which align based on pairwise probabilities between states only. The MAP IT theory reveals that alignment based on a projective divergence avoids normalization of weights (to obtain true probabilities) entirely, and further reveals a dual viewpoint via continuous densities and kernel smoothing. MAP IT is shown to produce visualizations which capture class structure better than the current state of the art.
Trosten, Daniel Johansen; Løkse, Sigurd Eivindson; Jenssen, Robert og Kampffmeyer, Michael Christian. (2024).
Leveraging tensor kernels to reduce objective function mismatch in deep clustering.
Vis sammendrag
Objective Function Mismatch (OFM) occurs when the optimization of one objective has a negative impact on the optimization of another objective. In this work we study OFM in deep clustering, and find that the popular autoencoder-based approach to deep clustering can lead to both reduced clustering performance, and a significant amount of OFM between the reconstruction and clustering objectives. To reduce the mismatch, while maintaining the structure-preserving property of an auxiliary objective, we propose a set of new auxiliary objectives for deep clustering, referred to as the Unsupervised Companion Objectives (UCOs). The UCOs rely on a kernel function to formulate a clustering objective on intermediate representations in the network. Generally, intermediate representations can include other dimensions, for instance spatial or temporal, in addition to the feature dimension. We therefore argue that the naïve approach of vectorizing and applying a vector kernel is suboptimal for such representations, as it ignores the information contained in the other dimensions. To address this drawback, we equip the UCOs with structure-exploiting tensor kernels, designed for tensors of arbitrary rank. The UCOs can thus be adapted to a broad class of network architectures. We also propose a novel, regression-based measure of OFM, allowing us to accurately quantify the amount of OFM observed during training. Our experiments show that the OFM between the UCOs and the main clustering objective is lower, compared to a similar autoencoder-based model. Further, we illustrate that the UCOs improve the clustering performance of the model, in contrast to the autoencoder-based approach. The code for our experiments is available at https://github.com/danieltrosten/tk-uco.
Uebbing, Lars; Joakimsen, Harald Lykke; Luppino, Luigi Tommaso; Martinsen, Iver; McDonald, Andrew; Wickstrøm, Kristoffer Knutsen; Lefevre, Sebastien Francois; Salberg, Arnt Børre; Hosking, Scott og Jenssen, Robert. (2024).
Investigating the Impact of Feature Reduction for Deep Learning-based Seasonal Sea Ice Forecasting.
NVA
poster
Jenssen, Robert; Lindsetmo, Rolf Ole; Blomsø, Therese Hoseth; Hofvind, Solveig Sand-Hanssen; Røed, Even A.; Hauglid, Mathias K. og Andreassen, Rune Nordgård. (2024).
Hvordan implementerer vi KI for bruk i helsesektoren på en trygg måte? UiT Norges arktiske universitet
NVA
Faglig foredrag
Choi, Changkyu; Yu, Shujian; Kampffmeyer, Michael Christian; Salberg, Arnt-Børre; Handegard, Nils Olav og Jenssen, Robert. (2024).
DIB-X: Formulating Explainability Principles for a Self-Explainable Model Through Information Theoretic Learning.
Vis sammendrag
The recent development of self-explainable deep learning approaches has focused on integrating well-defined explainability principles into learning process, with the goal of achieving these principles through optimization. In this work, we propose DIB-X, a self-explainable deep learning approach for image data, which adheres to the principles of minimal, sufficient, and interactive explanations. The minimality and sufficiency principles are rooted from the trade-off relationship within the information bottleneck framework. Distinctly, DIB-X directly quantifies the minimality principle using the recently proposed matrix-based Rényi’s α-order entropy functional, circumventing the need for variational approximation and distributional assumption. The interactivity principle is realized by incorporating existing domain knowledge as prior explanations, fostering explanations that align with established domain understanding. Empirical results on MNIST and two marine environment monitoring datasets with different modalities reveal that our approach primarily provides improved explainability with the added advantage of enhanced classification performance.
Jenssen, Robert; Eikvil, Line; Solberg, Anne H Schistad og Solheim, Inger. (2023).
Visual Intelligence Annual Report 2022.
Hauglid, Mathias K.; Eidum, Espen Morten Viklem og Jenssen, Robert. (2023).
Kunstig intelligens: Advarer mot diskriminering av minoriteter.
NVA
Populærvitenskapelig artikkel
Mancisidor, Rogelio Andrade; Kampffmeyer, Michael Christian; Aas, Kjersti og Jenssen, Robert. (2023).
Discriminative multimodal learning via conditional priors in generative models.
Vis sammendrag
Deep generative models with latent variables have been used lately to learn joint representations and generative processes from multi-modal data, which depict an object from different viewpoints. These two learning mechanisms can, however, conflict with each other and representations can fail to embed information on the data modalities. This research studies the realistic scenario in which all modalities and class labels are available for model training, e.g. images or handwriting, but where some modalities and labels required for downstream tasks are missing, e.g. text or annotations. We show, in this scenario, that the variational lower bound limits mutual information between joint representations and missing modalities. We, to counteract these problems, introduce a novel conditional multi-modal discriminative model that uses an informative prior distribution and optimizes a likelihood-free objective function that maximizes mutual information between joint representations and missing modalities. Extensive experimentation demonstrates the benefits of our proposed model, empirical results show that our model achieves state-of-the-art results in representative problems such as downstream classification, acoustic inversion, and image and annotation generation.
Olesen, Kristoffer Vinther; Boubekki, Ahcene; Kampffmeyer, Michael Christian; Jenssen, Robert; Christensen, Anders Nymark; Hørlück, Sune og Clemmensen, Line H.. (2023).
A Contextually Supported Abnormality Detector for Maritime Trajectories.
Vis sammendrag
The analysis of maritime traffic patterns for safety and security purposes is increasing in importance and, hence, Vessel Traffic Service operators need efficient and contextualized tools for the detection of abnormal maritime behavior. Current models lack interpretability and contextualization of their predictions and are generally not quantitatively evaluated on a large annotated dataset comprising all expected traffic in a Region of Interest. We propose a model for the detection of abnormal maritime behaviors that provides the closest behaviors as context to the predictions. The normalcy model relies on two-step clustering, which is first computed based on the positions of the vessels and then refined based on their kinematics. We design for each step a similarity measure, which combined are able to distinguish boats cruising shipping lanes in different directions, but also vessels with more freedom, such as pilot boats. Our proposed abnormality detection model achieved, on a large annotated dataset extracted from AIS logs that we publish, an ROC-AUC of 0.79, which is on a par with State-of-the-Art deep neural networks, while being more computationally efficient and more interpretable, thanks to the contextualization offered by our two-step clustering.
Blazquez-Garcia, Ane; Wickstrøm, Kristoffer Knutsen; Yu, Shujian; Mikalsen, Karl Øyvind; Boubekki, Ahcene; Conde, Angel; Mori, Usue; Jenssen, Robert og Lozano, Jose A.. (2023).
Selective Imputation for Multivariate Time Series Datasets with Missing Values.
Vis sammendrag
Multivariate time series often contain missing values for reasons such as failures in data collection mechanisms. Since these missing values can complicate the analysis of time series data, imputation techniques are typically used to deal with this issue. However, the quality of the imputation directly affects the performance of downstream tasks. In this paper, we propose a selective imputation method that identifies a subset of timesteps with missing values to impute in a multivariate time series dataset. This selection, which will result in shorter and simpler time series, is based on both reducing the uncertainty of the imputations and representing the original time series as good as possible. In particular, the method uses multi-objective optimization techniques to select the optimal set of points, and in this selection process, we leverage the beneficial properties of the Multi-task Gaussian Process (MGP). The method is applied to different datasets to analyze the quality of the imputations and the performance obtained in downstream tasks, such as classification or anomaly detection. The results show that much shorter and simpler time series are able to maintain or even improve both the quality of the imputations and the performance of the downstream tasks.
Eidum, Espen Morten Viklem; Hauglid, Mathias K. og Jenssen, Robert. (2023).
Kunstig intelligens: Forsker advarer mot diskriminering av minoriteter.
NVA
Populærvitenskapelig artikkel
Choi, Changkyu; Kampffmeyer, Michael; Handegard, Nils Olav; Salberg, Arnt-Børre og Jenssen, Robert. (2023).
Deep Semisupervised Semantic Segmentation in Multifrequency Echosounder Data.
Vis sammendrag
Multi-frequency echosounder data can provide a broad understanding of the underwater environment in a non-invasive manner. The analysis of echosounder data is, hence, a topic of great importance for the marine ecosystem. Semantic segmentation, a deep learning based analysis method predicting the class attribute of each acoustic intensity, has recently been in the spotlight of the fisheries and aquatic industry since its result can be used to estimate the abundance of the marine organisms. However, a fundamental problem with current methods is the massive reliance on the availability of large amounts of annotated training data, which can only be acquired through expensive handcrafted annotation processes, making such approaches unrealistic in practice. As a solution to this challenge, we propose a novel approach, where we leverage a small amount of annotated data (supervised deep learning) and a large amount of readily available unannotated data (unsupervised learning), yielding a new data-efficient and accurate semi-supervised semantic segmentation method, all embodied into a single end-to-end trainable convolutional neural networks architecture. Our method is evaluated on representative data from a sandeel survey in the North Sea conducted by the Norwegian Institute of Marine Research. The rigorous experiments validate that our method achieves comparable results utilizing only 40 percent of the annotated data on which the supervised method is trained, by leveraging unannotated data. The code is available at https://github.com/SFI-Visual-Intelligence/PredKlus-semisup-segmentation.
Østmo, Eirik Agnalt; Wickstrøm, Kristoffer; Radiya, Keyur; Kampffmeyer, Michael og Jenssen, Robert. (2023).
View it like a radiologist: Shifted windows for deep learning augmentation of CT images.
Vis sammendrag
Deep learning has the potential to revolutionize medical practice by automating and performing important tasks like detecting and delineating the size and locations of cancers in medical images. However, most deep learning models rely on augmentation techniques that treat medical images as natural images. For contrast-enhanced Computed Tomography (CT) images in particular, the signals producing the voxel intensities have physical meaning, which is lost during preprocessing and augmentation when treating such images as natural images. To address this, we propose a novel preprocessing and intensity augmentation scheme inspired by how radiologists leverage multiple viewing windows when evaluating CT images. Our proposed method, window shifting, randomly places the viewing windows around the region of interest during training. This approach improves liver lesion segmentation performance and robustness on images with poorly timed contrast agent. Our method outperforms classical intensity augmentations as well as the intensity augmentation pipeline of the popular nn-UNet on multiple datasets.
Choi, Changkyu; Kampffmeyer, Michael Christian; Handegard, Nils Olav; Salberg, Arnt-Børre og Jenssen, Robert. (2023).
Deep Semi-supervised Semantic Segmentation in Multi-frequency Echosounder Data. SFI Visual Intelligence
NVA
poster
Vis sammendrag
The fully supervised approaches achieve good performance provided that high-quality training data and an appropriate choice for the prediction model are assured. However, it is highly challenging for the echosounder data to obtain the class annotation for each backscattering intensity pixel because this relies on the manual annotation process, which is expensive and error-prone.
As a key step in this direction, we propose a novel deep semi-supervised semantic segmentation method that efficient- ly uses a small amount of manually annotated data by com- bining it with a large amount of readily available unannotated data in the learning process.
Poster presentation at VI days 2023. The original work is published in IEEE Journal of Oceanic Engineering DOI: 10.1109/JOE.2022.3226214.
Trosten, Daniel Johansen; Løkse, Sigurd Eivindson; Jenssen, Robert og Kampffmeyer, Michael. (2023).
On the Effects of Self-supervision and Contrastive Alignment in Deep Multi-view Clustering.
Vis sammendrag
Self-supervised learning is a central component in recent approaches to deep multi-view clustering (MVC). However, we find large variations in the development of self-supervision-based methods for deep MVC, potentially slowing the progress of the field. To address this, we present Deep-MVC, a unified framework for deep MVC that includes many recent methods as instances. We leverage our framework to make key observations about the effect of self-supervision, and in particular, drawbacks of aligning representations with contrastive learning. Further, we prove that contrastive alignment can negatively influence cluster separability, and that this effect becomes worse when the number of views increases. Motivated by our findings, we develop several new DeepMVC instances with new forms of self-supervision. We conduct extensive experiments and find that (i) in line with our theoretical findings, contrastive alignments decreases performance on datasets with many views; (ii) all methods benefit from some form of self-supervision; and (iii) our new instances outperform previous methods on several datasets. Based on our results, we suggest several promising directions for future research. To enhance the openness of the field, we provide an open-source implementation of DeepMVC, including recent models and our new instances. Our implementation includes a consistent evaluation protocol, facilitating fair and accurate evaluation of methods and components11Code: https://github.com/DanielTrosten/DeepMVC.
Trosten, Daniel Johansen; Chakraborty, Rwiddhi; Løkse, Sigurd Eivindson; Wickstrøm, Kristoffer; Jenssen, Robert og Kampffmeyer, Michael. (2023).
Hubs and Hyperspheres: Reducing Hubness and Improving Transductive Few-shot Learning with Hyperspherical Embeddings.
Vis sammendrag
Distance-based classification is frequently used in transductive few-shot learning (FSL). However, due to the high-dimensionality of image representations, FSL classifiers are prone to suffer from the hubness problem, where a few points (hubs) occur frequently in multiple nearest neighbour lists of other points. Hubness negatively impacts distance-based classification when hubs from one class appear often among the nearest neighbors of points from another class, degrading the classifier's performance. To address the hubness problem in FSL, we first prove that hubness can be eliminated by distributing representations uniformly on the hypersphere. We then propose two new approaches to embed representations on the hypersphere, which we prove optimize a tradeoff between uniformity and local similarity preservation - reducing hubness while retaining class structure. Our experiments show that the proposed methods reduce hubness, and significantly improves transductive FSL accuracy for a wide range of classifiers 11Code available at https://github.com/uitml/noHub..
Wickstrøm, Kristoffer; Østmo, Eirik Agnalt; Radiya, Keyur; Mikalsen, Karl Øyvind; Kampffmeyer, Michael og Jenssen, Robert. (2023).
A clinically motivated self-supervised approach for content-based image retrieval of CT liver images.
Vis sammendrag
Deep learning-based approaches for content-based image retrieval (CBIR) of computed tomography (CT) liver images is an active field of research but suffer from some critical limitations. First,they are heavily reliant on labelled data, which can be challenging and costly to acquire. Second, they lack transparency and explainability, which limits the trustworthiness of deep CBIR systems. We address these limitations by: (1) Proposing a self-supervised learning framework that incorporates domain-knowledge into the training procedure, and (2) by providing the first representation learning explainability analysis in the context of CBIR of CT liver images. Results demonstrate improved performance compared to the standard self-supervised approach across several metrics, as well as improved generalization across datasets. Further, we conduct the first representation learning explainability analysis in the context of CBIR, which reveals new insights into the feature extraction process. Lastly, we perform a case study with cross-examination CBIR that demonstrates the usability of our proposed framework. We believe that our proposed framework could play a vital role in creating trustworthy deep CBIR systems that can successfully take advantage of unlabeled data.
Wickstrøm, Kristoffer; Løkse, Sigurd Eivindson; Kampffmeyer, Michael; Yu, Shujian; Príncipe, José C. og Jenssen, Robert. (2023).
Analysis of Deep Convolutional Neural Networks Using Tensor Kernels and Matrix-Based Entropy.
Vis sammendrag
Analyzing deep neural networks (DNNs) via information plane (IP) theory has gained tremendous attention recently to gain insight into, among others, DNNs’ generalization ability. However, it is by no means obvious how to estimate the mutual information (MI) between each hidden layer and the input/desired output to construct the IP. For instance, hidden layers with many neurons require MI estimators with robustness toward the high dimensionality associated with such layers. MI estimators should also be able to handle convolutional layers while at the same time being computationally tractable to scale to large networks. Existing IP methods have not been able to study truly deep convolutional neural networks (CNNs). We propose an IP analysis using the new matrix-based Rényi’s entropy coupled with tensor kernels, leveraging the power of kernel methods to represent properties of the probability distribution independently of the dimensionality of the data. Our results shed new light on previous studies concerning small-scale DNNs using a completely new approach. We provide a comprehensive IP analysis of large-scale CNNs, investigating the different training phases and providing new insights into the training dynamics of large-scale neural networks.
Jenssen, Robert; Eikvil, Line; Solberg, Anne H Schistad og Solheim, Inger. (2022).
Visual Intelligence Annual Report 2021.
Vis sammendrag
The Visual Intelligence Annual Report 2021 highlights the centre's progress, activities and achieved innovations for 2021. It describes new deep learning methods which address pressing societal needs in the fields of medicine and health, marine science, the energy sector, and earth observation.
Wickstrøm, Kristoffer; Kampffmeyer, Michael; Mikalsen, Karl Øyvind og Jenssen, Robert. (2022).
Mixing up contrastive learning: Self-supervised representation learning for time series.
Vis sammendrag
The lack of labeled data is a key challenge for learning useful representation from time series data. However, an unsupervised representation framework that is capable of producing high quality representations could be of great value. It is key to enabling transfer learning, which is especially beneficial for medical applications, where there is an abundance of data but labeling is costly and time consuming. We propose an unsupervised contrastive learning framework that is motivated from the perspective of label smoothing. The proposed approach uses a novel contrastive loss that naturally exploits a data augmentation scheme in which new samples are generated by mixing two data samples with a mixing component. The task in the proposed framework is to predict the mixing component, which is utilized as soft targets in the loss function. Experiments demonstrate the framework’s superior performance compared to other representation learning approaches on both univariate and multivariate time series and illustrate its benefits for transfer learning for clinical time series.
Choi, Changkyu; Yu, Shujian; Kampffmeyer, Michael; Salberg, Arnt-Børre; Handegard, Nils Olav; Salahuddin, Suaiba Amina og Jenssen, Robert. (2022).
Explaining Marine Acoustic Target Classification in Multi-channel Echosounder Data using Self-attention Mask, Information-Bottleneck, and Mask Prior. Robert Jenssen, Inger Solheim
NVA
poster
Trosten, Daniel Johansen; Løkse, Sigurd Eivindson; Mikalsen, Karl Øyvind; Kampffmeyer, Michael og Jenssen, Robert. (2022).
RELAX: Representation Learning Explainability. UiT Machine Learning Group
NVA
poster
Trosten, Daniel Johansen; Wickstrøm, Kristoffer; Yu, Shujian; Løkse, Sigurd Eivindson; Jenssen, Robert og Kampffmeyer, Michael. (2022).
Deep Clustering with the Cauchy-Schwarz Divergence. Jose C. Principe, Robert Jenssen, Badong Chen, Shujian Yu
NVA
Vitenskapelig foredrag
Yu, Shujian; Alesiani, Francesco; Yin, Wenzhe; Jenssen, Robert og Principe, Jose C.. (2022).
Principle of Relevant Information for Graph Sparsification.
Mancisidor, Rogelio Andrade; Kampffmeyer, Michael; Aas, Kjersti og Jenssen, Robert. (2022).
Generating customer's credit behavior with deep generative models.
Vis sammendrag
Innovation is considered essential for today's organizations to survive and thrive. Researchers have also stressed the importance of leadership as a driver of followers' innovative work behavior (FIB). Yet, despite a large amount of research, three areas remain understudied: (a) The relative importance of different forms of leadership for FIB; (b) the mechanisms through which leadership impacts FIB; and (c) the degree to which relationships between leadership and FIB are generalizable across cultures. To address these lacunae, we propose an integrated model connecting four types of positive leadership behaviors, two types of identification (as mediating variables), and FIB. We tested our model in a global data set comprising responses of N = 7,225 participants from 23 countries, grouped into nine cultural clusters. Our results indicate that perceived LMX quality was the strongest relative predictor of FIB. Furthermore, the relationships between both perceived LMX quality and identity leadership with FIB were mediated by social identification. The indirect effect of LMX on FIB via social identification was stable across clusters, whereas the indirect effects of the other forms of leadership on FIB via social identification were stronger in countries high versus low on collectivism. Power distance did not influence the relations.
Liu, Qinghui; Kampffmeyer, Michael; Jenssen, Robert og Salberg, Arnt Børre. (2022).
Multi-modal land cover mapping of remote sensing images using pyramid attention and gated fusion networks.
Vis sammendrag
Multi-modality data is becoming readily available in remote sensing (RS) and can provide complementary information about the Earth’s surface. Effective fusion of multi-modal information is thus important for various applications in RS, but also very challenging due to large domain differences, noise, and redundancies. There is a lack of effective and scalable fusion techniques for bridging multiple modality encoders and fully exploiting complementary information. To this end, we propose a new multi-modality network (MultiModNet) for land cover mapping of multi-modal remote sensing data based on a novel pyramid attention fusion (PAF) module and a gated fusion unit (GFU). The PAF module is designed to efficiently obtain rich fine-grained contextual representations from each modality with a built-in cross-level and cross-view attention fusion mechanism, and the GFU module utilizes a novel gating mechanism for early merging of features, thereby diminishing hidden redundancies and noise. This enables supplementary modalities to effectively extract the most valuable and complementary information for late feature fusion. Extensive experiments on two representative RS benchmark datasets demonstrate the effectiveness, robustness, and superiority of the MultiModNet for multi-modal land cover classification.
Wickstrøm, Kristoffer; Johnson, Juan Emmanuel; Løkse, Sigurd Eivindson; Camps-Valls, Gusatu; Mikalsen, Karl Øyvind; Kampffmeyer, Michael og Jenssen, Robert. (2022).
The Kernelized Taylor Diagram.
Vis sammendrag
This paper presents the kernelized Taylor diagram, a graphical framework for visualizing similarities between data populations. The kernelized Taylor diagram builds on the widely used Taylor diagram, which is used to visualize similarities between populations. However, the Taylor diagram has several limitations such as not capturing non-linear relationships and sensitivity to outliers. To address such limitations, we propose the kernelized Taylor diagram. Our proposed kernelized Taylor diagram is capable of visualizing similarities between populations with minimal assumptions of the data distributions. The kernelized Taylor diagram relates the maximum mean discrepancy and the kernel mean embedding in a single diagram, a construction that, to the best of our knowledge, have not been devised prior to this work. We believe that the kernelized Taylor diagram can be a valuable tool in data visualization.
Jenssen, Robert; Eikvil, Line; Solberg, Anne H Schistad og Solheim, Inger. (2021).
Visual Intelligence Annual Report 2020.
Vis sammendrag
The Visual Intelligence Annual Report 2020 highlights the centre's progress, activities and achieved innovations for 2020. It describes new deep learning methods which address pressing societal needs in the fields of medicine and health, marine science, the energy sector, and earth observation.
Choi, Changkyu; Kampffmeyer, Michael; Handegard, Nils Olav; Salberg, Arnt Børre; Brautaset, Olav; Eikvil, Line og Jenssen, Robert. (2021).
Semi-supervised target classification in multi-frequency echosounder data.
Vis sammendrag
Acoustic target classification in multi-frequency echosounder data is a major interest for the marine ecosystem and fishery management since it can potentially estimate the abundance or biomass of the species. A key problem of current methods is the heavy dependence on the manual categorization of data samples. As a solution, we propose a novel semi-supervised deep learning method leveraging a few annotated data samples together with vast amounts of unannotated data samples, all in a single model. Specifically, two inter-connected objectives, namely, a clustering objective and a classification objective, optimize one shared convolutional neural network in an alternating manner. The clustering objective exploits the underlying structure of all data, both annotated and unannotated; the classification objective enforces a certain consistency to given classes using the few annotated data samples. We evaluate our classification method using echosounder data from the sandeel case study in the North Sea. In the semi-supervised setting with only a tenth of the training data annotated, our method achieves 67.6% accuracy, outperforming a conventional semi-supervised method by 7.0 percentage points. When applying the proposed method in a fully supervised setup, we achieve 74.7% accuracy, surpassing the standard supervised deep learning method by 4.7 percentage points.
Liu, Qinghui; Kampffmeyer, Michael; Jenssen, Robert og Salberg, Arnt Børre. (2021).
Self-constructing graph neural networks to model long-range pixel dependencies for semantic segmentation of remote sensing images.
Vis sammendrag
Capturing global contextual representations in remote sensing images by exploiting long-range pixel-pixel dependencies has been shown to improve segmentation performance.
However, how to do this efficiently is an open question as current approaches of utilising attention schemes, or very deep models to increase the field of view, increases complexity and memory consumption. Inspired by recent work on graph neural networks, we propose the Self-Constructing Graph (SCG) module that learns a long-range dependency graph directly from the image data and uses it to capture global contextual information efficiently to improve semantic segmentation. The SCG module provides a high degree of flexibility for constructing segmentation networks that seamlessly make use of the benefits of variants of graph neural networks (GNN) and convolutional neural networks (CNN). Our SCG-GCN model, a variant of SCG-Net built upon graph convolutional networks (GCN), performs semantic segmentation in an end-to-end manner with competitive performance on the publicly available ISPRS Potsdam and Vaihingen datasets, achieving a mean F1-scores of 92.0% and 89.8%, respectively. We conclude that the SCG-Net is an attractive architecture for semantic segmentation of remote sensing images since it achieves competitive performance with much fewer parameters and lower computational cost compared to related models based on convolutional neural networks.
Jenssen, Robert; Solberg, Anne H Schistad og Eikvil, Line. (2021).
Norge er verdensledende
på bildeanalyse med
kunstig intelligens.
NVA
Intervju
Løkse, Sigurd Eivindson; Mikalsen, Karl Øyvind; Kampffmeyer, Michael og Jenssen, Robert. (2021).
Towards Explainable Representation Learning. Norsk Forening for Bildebehandling og Maskinlæring
NVA
Faglig foredrag
Løkse, Sigurd Eivindson; Kampffmeyer, Michael; Jenssen, Robert og Mikalsen, Karl Øyvind. (2021).
Towards Explainable Representation Learning. Norsk Forening for Bildebehandling og Maskinlæring
NVA
Faglig foredrag
Handegard, Nils Olav; Eikvil, Line; Jenssen, Robert; Kampffmeyer, Michael; Salberg, Arnt Børre og Malde, Ketil. (2021).
Machine Learning + Marine Science: Critical Role of Partnerships in Norway.
Vis sammendrag
In this essay, we review some recent advances in developing machine learning (ML) methods for marine science applications in Norway. We focus mostly on deep learning (DL) methods and review the challenges we have faced in the process, including data preparation, (lack of) labelled training data, and interpretability. We also present the partnerships that have been formed between e-science institutions and marine science
institutions in Norway. These partnerships have been instrumental in moving this effort forward and have been fuelled by grants from the Norwegian Research Council. The last addition to this collaboration is the recent centres for research-based innovation in Marine Acoustic Abundance Estimation and Backscatter Classification (CRIMAC) and
Visual Intelligence (VI).
Liu, Qinghui; Kampffmeyer, Michael; Jenssen, Robert og Salberg, Arnt Børre. (2021).
PAGNet Models for The 2nd Agriculture-Vision Challenges CVPR 2021.
NVA
Vitenskapelig foredrag
Hansen, Stine; Gautam, Srishti; Jenssen, Robert og Kampffmeyer, Michael. (2021).
Anomaly Detection-Inspired Few-Shot Medical Image Segmentation Through Self-Supervision. Norsk Forening for Bildebehandling og Maskinlæring
NVA
Vitenskapelig foredrag
Jenssen, Robert. (2021).
Hvilken rolle kan kunstig intelligens ha i et kvinnehelseperspektiv? Kvinnehelseutvalget
NVA
Faglig foredrag
Choi, Changkyu; Kampffmeyer, Michael; Handegard, Nils Olav; Salberg, Arnt Børre; Eikvil, Line og Jenssen, Robert. (2021).
Semi-supervised Semantic Segmentation in Multi-frequency Echosounder Data. Robert Jenssen, Inger Solheim
NVA
poster
Jenssen, Robert og Moe, Trude Dagmar Haugseth. (2021).
Åpning av UiTs nye SFI: Visual Intelligence.
NVA
Populærvitenskapelig artikkel
Kampffmeyer, Michael; Jenssen, Robert; Mikalsen, Karl Øyvind og Løkse, Sigurd Eivindson. (2021).
Towards Explainable Representation Learning. Norsk Forening for Bildebehandling og Maskinlæring
NVA
Faglig foredrag
Liu, Qinghui; Kampffmeyer, Michael; Jenssen, Robert og Salberg, Arnt Børre. (2020).
SCG-Net for Semantic Labeling. IEEE GRSS
NVA
Vitenskapelig foredrag
Vis sammendrag
Graph Neural Networks (GNNs) have received increasing
attention in many fields. However, due to the lack of prior
graphs, their use for semantic labeling has been limited. Here, we propose a novel architecture called the Self-Constructing Graph (SCG), which makes use of learnable latent variables to generate embeddings and to self-construct the underlying graphs directly from the input features without relying on manually built prior knowledge graphs. SCG can automatically obtain optimized non-local context graphs from
complex-shaped objects in aerial imagery. We optimize SCG
via an adaptive diagonal enhancement method and a variational lower bound that consists of a customized graph reconstruction term and a Kullback-Leibler divergence regularization term. We demonstrate the effectiveness and flexibility of the proposed SCG on the publicly available ISPRS Vaihingen dataset and our model SCG-Net achieves competitive results in terms of F1-score with much fewer parameters and at a lower computational cost compared to related pure-CNN based work.
Liu, Qinghui; Kampffmeyer, Michael; Jenssen, Robert og Salberg, Arnt Børre. (2020).
Multi-View Self-Constructing Graph Convolutional Networks With Adaptive Class Weighting Loss for Semantic Segmentation.
Vis sammendrag
We propose a novel architecture called the Multi-view Self-Constructing Graph Convolutional Networks (MSCG-Net) for semantic segmentation. Building on the recently proposed Self-Constructing Graph (SCG) module, which makes use of learnable latent variables to self-construct the underlying graphs directly from the input features without relying on manually built prior knowledge graphs, we leverage multiple views in order to explicitly exploit the rotational invariance in airborne images. We further develop an adaptive class weighting loss to address the class imbalance. We demonstrate the effectiveness and flexibility of the proposed method on the Agriculture-Vision challenge dataset and our model achieves very competitive results (0.547 mIoU) with much fewer parameters and at a lower computational cost compared to related pure-CNN based work.
Mancisidor, Rogelio Andrade; Kampffmeyer, Michael; Aas, Kjersti og Jenssen, Robert. (2020).
Learning latent representations of bank customers with the Variational Autoencoder.
Vis sammendrag
Learning data representations that reflect the customers’ creditworthiness can improve marketing campaigns, customer relationship management, data and process management or the credit risk assessment in retail banks. In this research, we show that it is possible to steer data representations in the latent space of the Variational Autoencoder (VAE) using a semi-supervised learning framework and a specific grouping of the input data called Weight of Evidence (WoE). Our proposed method learns a latent representation of the data showing a well-defied clustering structure. The clustering structure captures the customers’ creditworthiness, which is unknown a priori and cannot be identified in the input space. The main advantages of our proposed method are that it captures the natural clustering of the data, suggests the number of clusters, captures the spatial coherence of customers’ creditworthiness, generates data representations of unseen customers and assign them to one of the existing clusters. Our empirical results, based on real data sets reflecting different market and economic conditions, show that none of the well-known data representation models in the benchmark analysis are able to obtain well-defined clustering structures like our proposed method. Further, we show how banks can use our proposed methodology to improve marketing campaigns and credit risk assessment.
Mancisidor, Rogelio Andrade; Kampffmeyer, Michael; Aas, Kjersti og Jenssen, Robert. (2020).
Deep generative models for reject inference in credit scoring.
Vis sammendrag
Credit scoring models based on accepted applications may be biased and their consequences can have a statistical and economic impact. Reject inference is the process of attempting to infer the creditworthiness status of the rejected applications. Inspired by the promising results of semi-supervised deep generative models, this research develops two novel Bayesian models for reject inference in credit scoring combining Gaussian mixtures and auxiliary variables in a semi-supervised framework with generative models. To the best of our knowledge this is the first study coupling these concepts together. The goal is to improve the classification accuracy in credit scoring models by adding reject applications. Further, our proposed models infer the unknown creditworthiness of the rejected applications by exact enumeration of the two possible outcomes of the loan (default or non-default). The efficient stochastic gradient optimization technique used in deep generative models makes our models suitable for large data sets. Finally, the experiments in this research show that our proposed models perform better than classical and alternative machine learning models for reject inference in credit scoring, and that model performance increases with the amount of data used for model training.
Liu, Qinghui; Kampffmeyer, Michael; Jenssen, Robert og Salberg, Arnt-Børre. (2020).
Self-Constructing Graph Convolutional Networks for Semantic Labeling.
Vis sammendrag
Graph Neural Networks (GNNs) have received increasing attention in many fields. However, due to the lack of prior graphs, their use for semantic labeling has been limited. Here, we propose a novel architecture called the Self-Constructing Graph (SCG), which makes use of learnable latent variables to generate embeddings and to self-construct the underlying graphs directly from the input features without relying on manually built prior knowledge graphs. SCG can automatically obtain optimized non-local context graphs from complex-shaped objects in aerial imagery. We optimize SCG via an adaptive diagonal enhancement method and a variational lower bound that consists of a customized graph reconstruction term and a Kullback-Leibler divergence regularization term. We demonstrate the effectiveness and flexibility of the proposed SCG on the publicly available ISPRS Vaihingen dataset and our model SCG-Net achieves competitive results in terms of F1-score with much fewer parameters and at a lower computational cost compared to related pure-CNN based work.
Chiu, Mang Tik; Xingqiang, Xu; Wang, Kai; Hobbs, Jennifer; Hovakimyan, Naira; Huang, Thomas S.; Shi, Honghui; Wei, Yunchao; Huang, Zilong; Schwing, Alexander; Brunner, Robert; Dozier, Ivan; Dozier, Wyatt; Ghandilyan, Karen; Wilson, David; Park, Hyunseong; Kim, Junhee; Kim, Sungho; Liu, Qinghui; Kampffmeyer, Michael; Jenssen, Robert; Salberg, Arnt Børre; Barbosa, Alexandre; Trevisan, Rodrigo; Zhao, Bingchen; Yu, Shaozuo; Yang, Siwei; Wang, Yin; Sheng, Hao; Chen, Xiao; Su, Jingyi; Rajagopal, Ram; Ng, Andrew; Huynh, Van Thong; Kim, Soo-Hyung; Na, In-Seop; Baid, Ujjwal; Innani, Shubham; Dutande, Prasad; Baheti, Bhakti; Talbar, Sanjay og Tang, Jianyu. (2020).
The 1st Agriculture-Vision Challenge: Methods and Results.
Vis sammendrag
The first Agriculture-Vision Challenge aims to encourage research in developing novel and effective algorithms for agricultural pattern recognition from aerial images, especially for the semantic segmentation task associated with our challenge dataset. Around 57 participating teams from various countries compete to achieve state-of-the-art in aerial agriculture semantic segmentation. The Agriculture-Vision Challenge Dataset was employed, which comprises of 21,061 aerial and multi-spectral farmland images. This paper provides a summary of notable methods and results in the challenge. Our submission server and leaderboard will continue to open for researchers that are interested in this challenge dataset and task; the link can be found here.
Liu, Qinghui; Kampffmeyer, Michael; Jenssen, Robert og Salberg, Arnt Børre. (2020).
MSCG-Net with Adaptive Class Weighting Loss for Semantic Segmentation. CVPR and AGRICULTURE-VISION
NVA
Vitenskapelig foredrag
Vis sammendrag
We propose a novel architecture called the Multi-view Self-Constructing Graph Convolutional Networks (MSCG-Net) for semantic segmentation. Building on the recently proposed Self-Constructing Graph (SCG) module, which makes use of learnable latent variables to self-construct the underlying graphs directly from the input features without relying on manually built prior knowledge graphs, we leverage multiple views in order to explicitly exploit the rotational invariance in airborne images. We further develop an adaptive class weighting loss to address the class imbalance. We demonstrate the effectiveness and flexibility of the proposed method on the Agriculture-Vision challenge dataset and our model achieves very competitive results
(0.547 mIoU) with much fewer parameters and at a lower
computational cost compared to related pure-CNN based
work.
Liu, Qinghui; Kampffmeyer, Michael; Jenssen, Robert og Salberg, Arnt Børre. (2020).
MSCG-Net Models for The 1st Agriculture-Vision Challenge CVPR 2020. CVPR and AGRICULTURE-VISION
NVA
Vitenskapelig foredrag
Liu, Qinghui; Kampffmeyer, Michael; Jenssen, Robert og Salberg, Arnt Børre. (2020).
Dense dilated convolutions merging network for land cover classification.
Vis sammendrag
Land cover classification of remote sensing images is a challenging task due to limited amounts of annotated data, highly imbalanced classes, frequent incorrect pixel-level annotations, and an inherent complexity in the semantic segmentation task. In this article, we propose a novel architecture called the dense dilated convolutions' merging network (DDCM-Net) to address this task. The proposed DDCM-Net consists of dense dilated image convolutions merged with varying dilation rates. This effectively utilizes rich combinations of dilated convolutions that enlarge the network's receptive fields with fewer parameters and features compared with the state-of-the-art approaches in the remote sensing domain. Importantly, DDCM-Net obtains fused local- and global-context information, in effect incorporating surrounding discriminative capability for multiscale and complex-shaped objects with similar color and textures in very high-resolution aerial imagery. We demonstrate the effectiveness, robustness, and flexibility of the proposed DDCM-Net on the publicly available ISPRS Potsdam and Vaihingen data sets, as well as the DeepGlobe land cover data set. Our single model, trained on three-band Potsdam and Vaihingen data sets, achieves better accuracy in terms of both mean intersection over union (mIoU) and F1-score compared with other published models trained with more than three-band data. We further validate our model on the DeepGlobe data set, achieving state-of-the-art result 56.2% mIoU with much fewer parameters and at a lower computational cost compared with related recent work.
Kampffmeyer, Michael; Jenssen, Robert; Mikalsen, Karl Øyvind og Revhaug, Arthur. (2020).
Uncertainty-Aware Deep Ensembles for Explainable Time Series Prediction. UiT Machine Learning Group
NVA
Faglig foredrag
Liu, Qinghui; Kampffmeyer, Michael; Jenssen, Robert og Salberg, Arnt Børre. (2019).
Road Mapping in Lidar Images Using a Joint-Task Dense Dilated Convolutions Merging Network.
Vis sammendrag
It is important, but challenging, for the forest industry to accurately map roads which are used for timber transport by trucks. In this work, we propose a Dense Dilated Convolutions Merging Network (DDCM-Net) to detect these roads in lidar images. The DDCM-Net can effectively recognize multi-scale and complex shaped roads with similar texture and colors, and also is shown to have superior performance over existing methods. To further improve its ability to accurately infer categories of roads,
Liu, Qinghui; Kampffmeyer, Michael C.; Jenssen, Robert og Salberg, Arnt Børre. (2019).
DDCM Network for Semantic Mapping of Remote Sensing Images. UiT Machine Learning Group
NVA
Vitenskapelig foredrag
Liu, Qinghui; Kampffmeyer, Michael C.; Jenssen, Robert og Salberg, Arnt Børre. (2019).
DDCM-Net for Semantic Mapping of Remote Sensing Images. IEEE GRSS and ISPRS
NVA
poster
Kampffmeyer, Michael C.; Løkse, Sigurd; Bianchi, Filippo Maria; Livi, Lorenzo; Salberg, Arnt Børre og Jenssen, Robert. (2019).
Deep divergence-based approach to clustering.
Vis sammendrag
A promising direction in deep learning research consists in learning representations and simultaneously discovering cluster structure in unlabeled data by optimizing a discriminative loss function. As opposed to supervised deep learning, this line of research is in its infancy, and how to design and optimize suitable loss functions to train deep neural networks for clustering is still an open question. Our contribution to this emerging field is a new deep clustering network that leverages the discriminative power of information-theoretic divergence measures, which have been shown to be effective in traditional clustering. We propose a novel loss function that incorporates geometric regularization constraints, thus avoiding degenerate structures of the resulting clustering partition. Experiments on synthetic benchmarks and real datasets show that the proposed network achieves competitive performance with respect to other state-of-the-art methods, scales well to large datasets, and does not require pre-training steps.
Bianchi, Filippo Maria; Livi, Lorenzo; Mikalsen, Karl Øyvind; Kampffmeyer, Michael C. og Jenssen, Robert. (2019).
Learning representations of multivariate time series with missing data.
Vis sammendrag
Learning compressed representations of multivariate time series (MTS) facilitates data analysis in the presence of noise and redundant information, and for a large number of variates and time steps. However, classical dimensionality reduction approaches are designed for vectorial data and cannot deal explicitly with missing values. In this work, we propose a novel autoencoder architecture based on recurrent neural networks to generate compressed representations of MTS. The proposed model can process inputs characterized by variable lengths and it is specifically designed to handle missing data. Our autoencoder learns fixed-length vectorial representations, whose pairwise similarities are aligned to a kernel function that operates in input space and that handles missing values. This allows to learn good representations, even in the presence of a significant amount of missing data. To show the effectiveness of the proposed approach, we evaluate the quality of the learned representations in several classification tasks, including those involving medical data, and we compare to other methods for dimensionality reduction. Successively, we design two frameworks based on the proposed architecture: one for imputing missing data and another for one-class classification. Finally, we analyze under what circumstances an autoencoder with recurrent layers can learn better compressed representations of MTS than feed-forward architectures.
Liu, Qinghui; Kampffmeyer, Michael C.; Jenssen, Robert og Salberg, Arnt Børre. (2019).
Road Mapping in Lidar Images Using a Joint-Task Dense Dilated Convolutions Merging Network. IEEE GRSS
NVA
Vitenskapelig foredrag
Liu, Qinghui; Kampffmeyer, Michael C.; Jenssen, Robert og Salberg, Arnt Børre. (2019).
Dense Dilated Convolutions Merging Network for Semantic Mapping of Remote Sensing Images.
Vis sammendrag
We propose a network for semantic mapping called the Dense Dilated Convolutions Merging Network (DDCM-Net) to provide a deep learning approach that can recognize multi-scale and complex shaped objects with similar color and textures, such as buildings, surfaces/roads, and trees in very high resolution remote sensing images. The proposed DDCM-Net consists of dense dilated convolutions merged with varying dilation rates. This can effectively enlarge the kernels' receptive fields, and, more importantly, obtain fused local and global context information to promote surrounding discriminative capability. We demonstrate the effectiveness of the proposed DDCM-Net on the publicly available ISPRS Potsdam dataset and achieve a performance of 92.3% F1-score and 86.0% mean intersection over union accuracy by only using the RGB bands, without any post-processing. We also show results on the ISPRS Vaihingen dataset, where the DDCM-Net trained with IRRG bands, also obtained better mapping accuracy (89.8% F1-score) than previous state-of-the-art approaches.
Liu, Qinghui; Salberg, Arnt Børre og Jenssen, Robert. (2018).
A Comparison of Deep Learning Architectures for Semantic Mapping of Very High Resolution Images.
Vis sammendrag
Semantic mapping of land cover is a key, but challenging, problem in remote sensing. Recent advances in deep learning, especially deep convolutional neural networks (CNNs), have shown outstanding performance in this task. In order to develop refined deep learning pipeline for meeting the rising need for accurate semantic mapping in remote sensing images, this paper study and compare a number of advanced deep learning segmentation architectures, which have obtained state-of-the-art results on computer vision contests like the Pascal VOC. To further analyze and compare the effectiveness of some elaborate layers and underlying structures introduced by these architectures, we evaluate them by re-implementing, train and test them on ISPRS Potsdam dataset. Our results show that a promising performance with overall Fl_score above 87% and mIoU of 79% can be obtained by only using the RGB images, without any post-processing such as conditional random field (CRF) smoothing. At last, we propose several possible approaches to further enhance the deep learning architectures to better deal with high-resolution aerial images. We therefore consider this work to be helpful for the remote sensing research community.
Liu, Qinghui; Salberg, Arnt Børre og Jenssen, Robert. (2018).
A COMPARISON OF DEEP LEARNING ARCHITECTURES FOR SEMANTIC MAPPING OF VERY HIGH RESOLUTION IMAGES. IGARSS 2018
NVA
poster
Vis sammendrag
Semantic mapping of land cover is a key, but challenging, problem in remote sensing. Recent advances in deep learning, especially deep convolutional neural networks (CNNs), have shown outstanding performance in this task. In order to develop refined deep learning pipeline for meeting the rising need for accurate semantic mapping in remote sensing images, this paper study and compare a number of advanced deep learning segmentation architectures, which have obtained state-of-the-art results on computer vision contests like the Pascal VOC. To further analyze and compare the effectiveness of some elaborate layers and underlying structures introduced by these architectures, we evaluate them by re-implementing, train and test them on ISPRS Potsdam dataset. Our results show that a promising performance with overall Fl_score above 87% and mIoU of 79% can be obtained by only using the RGB images, without any post-processing such as conditional random field (CRF) smoothing. At last, we propose several possible approaches to further enhance the deep learning architectures to better deal with high-resolution aerial images. We therefore consider this work to be helpful for the remote sensing research community.
Kampffmeyer, Michael C.; Salberg, Arnt Børre og Jenssen, Robert. (2018).
Urban land cover classification with missing data modalities using deep convolutional neural networks.
Vis sammendrag
Automatic urban land cover classification is a fundamental problem in remote sensing, e.g., for environmental monitoring. The problem is highly challenging, as classes generally have high interclass and low intraclass variances. Techniques to improve urban land cover classification performance in remote sensing include fusion of data from different sensors with different data modalities. However, such techniques require all modalities to be available to the classifier in the decision-making process, i.e., at test time, as well as in training. If a data modality is missing at test time, current state-of-the-art approaches have in general no procedure available for exploiting information from these modalities. This represents a waste of potentially useful information. We propose as a remedy a convolutional neural network (CNN) architecture for urban land cover classification which is able to embed all available training modalities in the so-called hallucination network. The network will in effect replace missing data modalities in the test phase, enabling fusion capabilities even when data modalities are missing in testing. We demonstrate the method using two datasets consisting of optical and digital surface model (DSM) images. We simulate missing modalities by assuming that DSM images are missing during testing. Our method outperforms both standard CNNs trained only on optical images as well as an ensemble of two standard CNNs. We further evaluate the potential of our method to handle situations where only some DSM images are missing during testing. Overall, we show that we can clearly exploit training time information of the missing modality during testing.
Kampffmeyer, Michael C.; Løkse, Sigurd; Bianchi, Filippo Maria; Jenssen, Robert og Livi, Lorenzo. (2018).
The deep kernelized autoencoder.
Vis sammendrag
Autoencoders learn data representations (codes) in such a way that the input is reproduced at the output of the network. However, it is not always clear what kind of properties of the input data need to be captured by the codes. Kernel machines have experienced great success by operating via inner-products in a theoretically well-defined reproducing kernel Hilbert space, hence capturing topological properties of input data. In this paper, we enhance the autoencoder's ability to learn effective data representations by aligning inner products between codes with respect to a kernel matrix. By doing so, the proposed kernelized autoencoder allows learning similarity-preserving embeddings of input data, where the notion of similarity is explicitly controlled by the user and encoded in a positive semi-definite kernel matrix. Experiments are performed for evaluating both reconstruction and kernel alignment performance in classification tasks and visualization of high-dimensional data. Additionally, we show that our method is capable to emulate kernel principal component analysis on a denoising task, obtaining competitive results at a much lower computational cost.
Kampffmeyer, Michael C.; Løkse, Sigurd; Bianchi, Filippo Maria; Livi, Lorenzo; Salberg, Arnt Børre og Jenssen, Robert. (2017).
Deep divergence-based clustering.
Vis sammendrag
A promising direction in deep learning research is to learn representations and simultaneously discover cluster structure in unlabeled data by optimizing a discriminative loss function. Contrary to supervised deep learning, this line of research is in its infancy and the design and optimization of a suitable loss function with the aim of training deep neural networks for clustering is still an open challenge. In this paper, we propose to leverage the discriminative power of information theoretic divergence measures, which have experienced success in traditional clustering, to develop a new deep clustering network. Our proposed loss function incorporates explicitly the geometry of the output space, and facilitates fully unsupervised training end-to-end. Experiments on real datasets show that the proposed algorithm achieves competitive performance with respect to other state-of-the-art methods.
Kampffmeyer, Michael C.; Salberg, Arnt Børre og Jenssen, Robert. (2017).
Urban land cover classification with missing data using deep convolutional neural networks. IEEE Geoscience and Remote Sensing Society
NVA
Vitenskapelig foredrag
Kampffmeyer, Michael C.; Salberg, Arnt Børre og Jenssen, Robert. (2017).
Urban land cover classification with missing data using deep convolutional neural networks.
Vis sammendrag
Fusing different sensors with different data modalities is a common technique to improve land cover classification performance in remote sensing. However, all modalities are rarely available for all test data, and this missing data problem poses severe challenges for multi-modal learning. Inspired by recent successes in deep learning, we propose as a remedy a convolutional neural network architecture for urban remote sensing image segmentation trained on data modalities which are not all available at test time. We train our architecture with a cost function particularly suited for imbalanced classes, as this is a frequent problem in remote sensing. We demonstrate the method using a benchmark dataset containing RGB and DSM images. Assuming that the DSM images are missing during testing, our method outperforms both a CNN trained on RGB images as well as an ensemble of two CNNs trained on the RGB images, by exploiting the training time information of the missing modality.