Arnt-Børre Salberg

Seabirds are among the most threatened avian taxa globally, with over half of all species in decline. Accurate population estimates are essential for tracking trends and informing conservation, yet traditional survey methods are limited by logistical challenges, high costs, and the potential for wildlife disturbance, particularly in remote coastal areas. Unoccupied aerial vehicles (UAVs or drones) offer an efficient and low-disturbance alternative, but the vast volumes of imagery they produce are often labour-intensive to analyse. In this study, we combined drone imagery with deep learning techniques to estimate colony size and abundance of surface-nesting seabirds based on counts of visible individuals. High-resolution aerial imagery was collected from 163 colonies along the southern and central Norwegian coastline over three breeding seasons (2022–2024), covering a total of 7.67 km2. A convolutional neural network (Faster R-CNN with ResNet-101 backbone) was trained on 131 annotated orthomosaics and evaluated on 32 additional annotated orthomosaics from geographically distinct colonies. Across all data, 23,062 individual seabirds were annotated. Colonies hosted an average of 141.5 ± 193.9 individuals and 4.1 ± 2.3 focal species per site. At a confidence threshold of 0.7, the model achieved a detection rate of 87.5 % and a macro F1-score of 0.88. It performed well across multiple focal species, including terns, gulls, and cormorants, and remained robust in mixed-species colonies. Most errors involved false negatives or confusion among visually similar species. Our results demonstrate the potential for deep learning models to support efficient, scalable, and low-disturbance seabird monitoring across diverse habitats, reducing manual annotation effort and informing conservation practice.

The escalating demand for high-resolution Earth Observation (EO) data for various applications has significantly influenced advancements in image processing techniques. This study proposes a workflow to super-resolve the 12 spectral bands of Sentinel-2 Level-2A imagery to a ground sampling distance of 2.5m. The method leverages a hybrid approach, integrating advanced diffusion models with image fusion techniques. A critical component of the proposed methodology is super-resolution of Sentinel-2 RGB bands to generate a super-resolved Sentinel-2 RGB image, which subsequently serves in the image fusion pipeline that super-resolves the remaining spectral bands. The super-resolution algorithm is based on a diffusion model and is trained using the extensive National Agriculture Imagery Program (NAIP) dataset of aerial images, which is freely available. To make the super-resolution algorithm, trained on NAIP images, applicable to Sentinel-2 imagery, image harmonization and degradation were necessary to compensate for the inherent differences between NAIP and Sentinel-2 imagery. To address this challenge, we utilised a sophisticated degradation and harmonisation model that accurately simulates Sentinel-2 images from NAIP data, ensuring the harmonised NAIP images closely mimic the characteristics of Sentinel-2 observations post-resolution reduction. To investigate if learning the diffusion model using a large dataset of airborne images like NAIP provides better results than learning the model using a smaller satellite-based dataset like WorldStrat of high-resolution SPOT images, we performed a comparative analysis. The results demonstrate that models trained with the harmonised and correctly simulated datasets like NAIP significantly outperform those trained directly on SPOT images but also other existing super-resolution models available. This finding reveals that learning with more data can be beneficial if the data is properly harmonised and degraded to match the Sentinel-2 images. We performed a comprehensive evaluation using the recently established open-SR test methodology to validate the proposed model across multiple super-resolution metrics. This testing framework rigorously evaluates the super-resolution model based on metrics beyond traditional super-resolution metrics such as PSNR, SSIM, and LPIPS. Instead, the open-SR test evaluates the model based on metrics that measure its consistency, synthesis, and correctness. The proposed super-resolution model outperformed several current state-of-the-art models based on the comprehensive open-SR test framework. In addition, visual comparison further established the superior performance of our model in both urban and rural scenarios. An important component of the proposed model is the super-resolution of all 12 Sentinel-2 Level-2A bands, contrary to previous work, which has mainly focused on RGB band super-resolution. The proposed fusion pipeline successfully utilises the super-resolved image to obtain an enhanced 12-band Sentinel 2 image, similar to pansharpening techniques. We show qualitative and quantitative results on all 12 bands that demonstrate the seamless performance of the fusion method in super-resolution. This study not only showcases the potential of combining AI-driven super-resolution models with image fusion techniques in enhancing EO data resolution but also addresses the critical challenges posed by the diversity in data sources and the necessity for accurate generative models in training neural networks for super-resolution tasks.

This report details the data analysis products developed within the SeaBee project, a national infrastructure for drone-based services for use in coastal and aquatic research, mapping and monitoring of habitats, animal communities, and anthropogenic impacts. We present an advanced, automated data analysis pipeline that leverages deep learning for two primary tasks: pixel-wise thematic mapping of coastal habitats and object detection for counting wildlife. The pipeline utilizes models such as U-Net and Faster R-CNN to process high-resolution drone imagery (RGB, MSI, and HSI) and incorporates a novel hierarchical classification structure for habitat mapping and a robust method for detecting out-of-distribution (OOD) samples. We demonstrate the pipeline's pixel-wise mapping effectiveness through extensive experiments at three diverse Norwegian coastal sites—Remøy, Vega, and Ølbergholmen—achieving high accuracy in mapping complex habitats like kelp forests and various substrate types. Furthermore, the object detection framework shows strong performance in the automated counting and classification of 11 seabird species and coastal seals, offering a significant improvement in efficiency over traditional survey methods. The results confirm that the SeaBee pipeline is a powerful, scalable tool for environmental research and management, though we also discuss challenges such as data imbalance and model generalizability that will inform future work. This research is funded by the Research Council of Norway, project ID #296478, to the Norwegian Infrastructure for drone-based research, mapping, and monitoring in the coastal zone (SeaBee).

Foundation models (FMs) are transforming the field of Artificial Intelligence (AI) by learning inherent information from vast amounts of unlabeled data, enabling adaptation to numerous applications. Their integration into the Earth Observation (EO) ecosystem promises to revolutionize the information value chain, impacting industry, research, and science. However, EO applications present unique challenges, including the diverse needs for detail or rapid processing, and the variety of data value and sensor characteristics. Models must handle data from multiple sensors at varying ground sampling distances (GSD). Vision Transformers (ViT), often trained using self-supervised learning (SSL), form the backbone of many modern FMs by learning from complex data patterns without explicit supervision. We introduce FM4CS, a versatile foundation model specifically designed for climate and society EO applications. FM4CS aims to address the aforementioned challenges by supporting four different Sentinel sensors: Sentinel-1 SAR, Sentinel-2 MSI, Sentinel-3 OLCI, and Sentinel-3 SLSTR. Inspired by approaches like USat for multi-sensor data handling at native resolutions and FlexiViT for operating across a wide range of patch sizes without retraining, FM4CS employs a single ViT architecture. The model utilizes individual patch embedding layers for each sensor channel, allowing flexibility in processing subsets of spectral bands. It adapts the number of patches per band based on GSD and uses spectral group pooling to manage token sequence length. To accommodate flexible patch sizes, FM4CS incorporates a training procedure where patch size is randomized, allowing the model to adapt dynamically at inference time. For handling the need for positional information of the patches across different sensors, resolutions and patch sizes, FM4CS adapts the 2D ALiBi (Attention by Linear Bias) relative positional encoding scheme. The pre-training dataset for FM4CS is curated to ensure diversity. Instead of stacking small image crops, data is sampled using the Sentinel-2 tiling grid, co-locating Sentinel-1, Sentinel-2, and Sentinel-3 imagery for given locations and time intervals. A stratified sampling approach, based on k-means clustering of ESA WorldCover maps and Sentinel-2 RGB composites, is used to capture the diversity of global land cover and address imbalances. Oceanic data sampling incorporates shipping traffic density, oil and gas installations, and areas with higher probabilities of sea ice and icebergs. The dataset also includes ERA5-Land variables to facilitate multi-modal pretext tasks, leveraging daily statistics for variables such as soil moisture, temperature, and snow cover. FM4CS is trained using several SSL tasks. These include pixel-level input band reconstruction, similar to masked image modeling, where a lightweight ViT decoder reconstructs masked tokens. Additionally, the model predicts existing maps such as ESA WorldCover from Sentinel-1/2 data, and other land cover maps (ESA GlobCover, MOD12Q1) from the Sentinel-3 sensors using a cross-entropy loss. Image-level tasks involve the prediction of ERA5 variables, latitude, longitude and data acquisition month.

To leverage Earth observation (EO) data for large scale analysis, automatic methods is a prerequisite. Since 2012, deep learning (DL) models have brought about a revolutionary change in the analysis of image data and are currently considered state-of-the-art for a broad spectrum of EO tasks. However, a bottleneck with supervised DL models is that they often require a vast amount of labelled data to be trained, and the research community has therefore started to explore alternatives to supervised learning. During the last years, foundation models (FM) signify a change of thinking in computer vision. FMs are trained on a vast volume of unlabeled data and can identify complex patterns due to their large-scale learning capabilities. Typically, an additional head or decoder (small network) is added to the FM, which is trained and adapted to various use-cases by means of a small amount of labelled data. FM have also started to be explored for EO applications, however, current EO-based FMs are limited in terms of handling different modalities with large differences in resolution. Modern FMs are often based on transformers and are trained using self-supervised learning (SSL). There are several SSL schemes in place, including masked autoencoders (MAE) where we mask part of the input data and learn the model to predict the masked data. This is not useful by itself, but the model learns compressed representation of the data, which can be leveraged in downstream applications. This potentially makes the FM more useful than models trained on a limited set of labeled data. The Norwegian Computing Center and UiT – The Arctic University of Norway are in collaboration with user partners Romanian National Meteorological Administration, Danish Meteorological Institute, Polar View and Norwegian Water Resources and Energy Directorate developing a multi-modal FM. The FM is designed to process data from the satellites Sentinel-1 SAR, Sentinel-2 and Sentinel-3 OLCI and SLSTR. The FM is based on vison transformers (ViT) but utilizing the same principle as the USat approach to handle the different resolutions between the modalities. The training of the FM is based on the MAE approach, and to ensure that the SSL work efficiently, we have developed a smart sampling scheme during that provides relevant and diverse training data. In addition to SSL, we have also created a learning task in regressing climate variables from the ERA5 dataset. To train the FM, over 20 TB of Sentinel data was collected and processed using the LUMI supercomputer. The multi-modal FM is demonstrated on the following use-cases: mapping of snow, flood zone mapping, mapping and monitoring of sea ice, iceberg detection, early draught warning and mapping of wetlands. The resolution of the target use-case products are vastly different, e.g. for snow mapping we aim for a ground sampling distance (GSD) of 250m whereas for flood zone mapping we aim for a GSD of 10m. We have therefore trained two versions of the FM: one aiming for high-resolution products with GSD between 10 – 60m, and one aiming for low-resolutions products with GSD above 100m. The downstream tasks are implemented using the open-source framework TerraTorch, which is a flexible fine-tuning framework for geospatial FMs. TerraTorch supports common fine-tuning tasks such as image segmentation and pixel-wise regression along with a selection of task-specific decoder heads.

To leverage Earth observation (EO) data for large scale analysis, automatic methods is a prerequisite. Since 2012, deep learning (DL) models have brought about a revolutionary change in the analysis of image data and are currently considered state-of-the-art for a broad spectrum of EO tasks. However, a bottleneck with supervised DL models is that they often require a vast amount of labelled data to be trained, and the research community has therefore started to explore alternatives to supervised learning. Inspired by the progress in large language models, foundation models (FM) are now being applied extensively in computer vision. FMs are trained on a vast volume of unlabeled data and can identify complex patterns due to their large-scale learning capabilities. Typically, an additional head or decoder (small network) is added to the FM, which is trained and adapted to various use-cases by means of a small amount of labelled data. FM have also started to be explored for EO applications, however, current EO-based FMs are limited in terms of handling different modalities with large differences in resolution. Modern FMs are often based on transformers and are trained using self-supervised learning (SSL). There are several SSL schemes in place, including masked autoencoders (MAE) where we mask part of the input data and learn the model to predict the masked data. This is not useful by itself, but the model learns compressed representation of the data, which can be leveraged in downstream applications. This potentially makes the FM more useful than models trained on a limited set of labeled data. The Norwegian Computing Center and UiT – The Arctic University of Norway are in collaboration with user partners Romanian National Meteorological Administration, Danish Meteorological Institute, Polar View and Norwegian Water Resources and Energy Directorate developing a multi-modal FM. The FM is designed to process data from the satellites Sentinel-1 SAR, Sentinel-2 and Sentinel-3 OLCI and SLSTR. The FM is based on vison transformers (ViT) but utilizing the same principle as the USat approach to handle the different resolutions between the modalities. The training of the FM is based on the MAE approach, and to ensure that the SSL work efficiently, we have developed a smart sampling scheme during that provides relevant and diverse training data. In addition to SSL, we have also created a learning task in regressing climate variables from the ERA5 dataset. To train the FM, over 20 TB of Sentinel data was collected and processed using the LUMI supercomputer. The multi-modal FM is demonstrated on the following use-cases: mapping of snow, flood zone mapping, mapping and monitoring of sea ice, iceberg detection, early draught warning and mapping of wetlands. The downstream tasks are implemented using the open-source framework TerraTorch, which is a flexible fine-tuning framework for geospatial FMs. The FM4CS model is one of the first to apply Sentinel-3 data, which makes it attractive for climate applications.

With the state-of-the-art IceNet model, deep learning has contributed to an important aspect of climate research by leveraging a range of climate inputs to provide accurate forecasts of Arctic sea ice concentration (SIC). The deep learning subfield of eXplainable AI (XAI) has gained enormous attention in order to gauge feature importance of neural networks, for instance by leveraging network gradients. In recent work, an XAI study of the IceNet was conducted, using gradient saliency maps to interrogate its feature importance. A majority of XAI studies provide information about feature importance as revealed by the XAI method, but rarely provide thorough analysis of effects from reducing the number of input variables. In this paper, we train versions of the IceNet with drastically reduced numbers of input features according to results of XAI and investigate the effects on the sea ice predictions, on average and with respect to specific events. Our results provide evidence that the model generally performs better when less features are used, but in case of anomalous events, a larger number of features is beneficial. We believe our thorough study of the IceNet in terms of feature importance revealed by XAI may give inspiration for other deep learning-based problem scenarios and application domains.

Accurate uncertainty information associated with essential climate variables (ECVs) is crucial for reliable climate modeling and understanding the spatiotemporal evolution of the Earth system. In recent years, geoscience and climate scientists have benefited from rapid progress in deep learning to advance the estimation of ECV products with improved accuracy. However, the quantification of uncertainties associated with the output of such deep learning models has yet to be thoroughly adopted. This survey explores the types of uncertainties associated with ECVs estimated from deep learning and the techniques to quantify them. The focus is on highlighting the importance of quantifying uncertainties inherent in ECV estimates, considering the dynamic and multifaceted nature of climate data. The survey starts by clarifying the definition of aleatoric and epistemic uncertainties and their roles in a typical satellite observation processing workflow, followed by bridging the gap between conventional statistical and deep learning views on uncertainties. Then, we comprehensively review the existing techniques for quantifying uncertainties associated with deep learning algorithms, focusing on their application in ECV studies. The specific need for modification to fit the requirements from both the Earth observation side and the deep learning side in such interdisciplinary tasks is discussed. Finally, we demonstrate our findings with two ECV examples, snow cover and terrestrial water storage, and provide our perspectives for future research.

Diffusion models have obtained photo-realistic results on various super-resolution tasks. However, existing approaches typically require the availability of high-resolution and paired training data, which often is not readily available in many remote sensing scenarios. To enhance multi-spectral Sentinel 2 (S2) satellite images – at a ground sampling distance (GSD) ranging from 10m to 60m – without requiring high-resolution or paired training data, we therefore propose and evaluate a novel set of approaches to leverage traditional pansharpening within a diffusion model context to simulate the required training data. We extensively compare the proposed methods and demonstrate that by utilizing unpaired Spot-6/7 data, we are able to produce photo-realistic S2 images at a resolution of 2.5m.

Aerial drone imaging is an efficient tool for mapping and monitoring of coastal habitats at high spatial and temporal resolution. Specifically, drone imaging allows for time- and cost-efficient mapping covering larger areas than traditional mapping and monitoring techniques, while also providing more detailed information than those from airplanes and satellites, enabling for example to differentiate various types of coastal vegetation. Here, we present a systematic method for shallow water habitat classification based on drone imagery. The method includes: • Collection of drone images and creation of orthomosaics. • Gathering ground-truth data in the field to guide the image annotation and to validate the final map product. • Annotation of drone images into – potentially hierarchical – habitat classes and training of machine learning algorithms for habitat classification. As a case study, we present a field campaign that employed these methods to map a coastal site dominated by seagrass, seaweed and kelp, in addition to sediments and rock. Such detailed but efficient mapping and classification can aid to understand and sustainably manage ecologically and valuable marine ecosystems.

The Norwegian Environment Agency is responsible for updating a map of undisturbed nature, which is performed every five years based on aerial photos. Some of the aerial photos are already up to five years old when a new version of the map of undisturbed nature is published. Thus, several new nature interventions may have been missed. To address this issue, the timeliness and mapping accuracy were improved by integrating Sentinel-2 satellite imagery for the detection of new roads across Norway. The focus on new roads was due to the fact that most new nature interventions include the construction of new roads. The proposed methodology is based on applying U-Net on all the available summer images with less than 10% cloud cover over a five-year period, with an aggregation step to summarize the predictions. The observed detection rate was 98%. Post-processing steps reduced the false positive rate to 46%. However, as the false positive rate was still substantial, the manual verification of the predicted new roads was needed. The false negative rate was low, except in areas without vegetation.

The KELPMAP study demonstrated that high-resolution multispectral data from drones and satellites, combined with AI-based image analysis, can efficiently map kelp forests and other coastal habitats. The field campaign, conducted in 2022 within Vega and Herøy municipalities, produced orthomosaics with 9 cm GSD for multispectral and 5 cm for RGB images. Based on drone data and AI, between 11% and 65% of the study area was identified as brown algae. Satellites overpredicted kelp forests but aligned with drone data after removing uncertain predictions. In Helgeland’s clear waters, benthic species and habitats were identified down to 10 meters. Using NIVA's statistical model, drones were estimated to map almost 60% of Norwegian kelp forests and 80-90% of total kelp biomass, despite only reaching 10 meters. Upscaling habitat maps using satellite images is possible but limited by satellite resolution. Drone-based training data enhances satellite-derived map accuracy. High-resolution drone maps are ideal for local marine spatial planning, while satellite maps are suitable for national level applications like carbon accounting. More ground truth data are needed for improved species-level mapping and validation of upscaled products. The study also assessed mapping benthic habitats according to NiN 3.0, identifying kelp forests, seaweed beds, eelgrass, and various seabed types.

The recent development of self-explainable deep learning approaches has focused on integrating well-defined explainability principles into learning process, with the goal of achieving these principles through optimization. In this work, we propose DIB-X, a self-explainable deep learning approach for image data, which adheres to the principles of minimal, sufficient, and interactive explanations. The minimality and sufficiency principles are rooted from the trade-off relationship within the information bottleneck framework. Distinctly, DIB-X directly quantifies the minimality principle using the recently proposed matrix-based Rényi’s α-order entropy functional, circumventing the need for variational approximation and distributional assumption. The interactivity principle is realized by incorporating existing domain knowledge as prior explanations, fostering explanations that align with established domain understanding. Empirical results on MNIST and two marine environment monitoring datasets with different modalities reveal that our approach primarily provides improved explainability with the added advantage of enhanced classification performance.

The main objective of the working group was to review recent surveys of Greenland Sea harp and hooded seal pup production and examine harvest scenarios for these populations as well as harp seals in the White Sea. No new survey to estimate pup production of Barents Sea/White Sea harp seals was completed. No new survey information was available for the Northwest Atlantic. The 2022 Greenland Sea aerial survey images were analyzed manually and with the aid of automatic detection methodology (deep learning). For assessment purposes, this report only refers to the manual counts. Correction factors based on staging surveys were applied according to established methodology. The 2022 Greenland Sea harp seal pup production estimate for harp seals was 92,769 (CV = 20.2%), which is significantly higher than the 2018 estimate but similar to that based on the 2012 survey. The hooded seal pup production estimate for 2022 was 13,509 (CV=12.9%), slightly but not significantly higher than the 2018 estimate. Subsequent to the recent benchmark meeting, model development indicated that the model estimates of adult population size for the Greenland Sea population of harp seals is highly sensitive to the standard deviation on the prior for initial population size. The WG therefore concluded that the current version of the assessment model could not be used to explore harvest scenarios based on estimates of current or projected total population size. Moreover, given the fact that the estimate of current total population size is unreliable, it also did not allow for robust calculation of Potential Biological removals (PBR). Tentatively, two different approaches are presented that might be used to inform sustainable harvest levels until the model has been further improved and reviewed: 1) an adaptive management approach based on population trends and 2) PBR based on a conservative population estimate that is a simple scaling of the observed levels of pup production, based on plausible values of adult:pup ratios. The Greenland Sea hooded seal population shows continued decline, and remains below the Lower Reference Limit despite no hunting since 2007. In a recent review of the status of the Northwest Atlantic harp seal population, model fit to aerial survey estimates of pup production and annual reproductive rates was poor compared to previous assessments indicating underlying problems relating to model assumptions and/or structure. A new hierarchical Bayesian state-space model was fitted to the same data on pup production, annual fecundity, human removals, and environmental conditions used in the previous assessment to produce annual estimates of pup production and total abundance from 1952 - 2019. Data on age structure based upon random samples were also included, and the process model incorporated environmental stochasticity and several other improvements. The new model estimates were similar to the previous model through 1990 but then diverged, indicating that the population peaked in 1997 at 6.6 million animals, almost a decade earlier than modelled in previous assessments. After a period of decline due to high catches and poor ice conditions, the new model provides an abundance estimate of 4.7 (95% Credibility Interval (CI) 3.7-5.7 ) million in 2019, compared to an estimate of 7.6 (95% CI 6.6-8.8) million in the last assessment. The lower estimates of recent abundance reflect higher and more variable juvenile mortality after 2000 due to a combination of density-dependent and density-independent factors operating on juvenile survival. The new model also suggests a decline in equilibrium abundance (K) levels from 7.6 (95% CI=7.4 to 7.8) million Northwest Atlantic harp seals prior to 2000 to 6.8 (95% CI=6.7 to 6.9) million animals post-2000.

Multi-frequency echosounder data can provide a broad understanding of the underwater environment in a non-invasive manner. The analysis of echosounder data is, hence, a topic of great importance for the marine ecosystem. Semantic segmentation, a deep learning based analysis method predicting the class attribute of each acoustic intensity, has recently been in the spotlight of the fisheries and aquatic industry since its result can be used to estimate the abundance of the marine organisms. However, a fundamental problem with current methods is the massive reliance on the availability of large amounts of annotated training data, which can only be acquired through expensive handcrafted annotation processes, making such approaches unrealistic in practice. As a solution to this challenge, we propose a novel approach, where we leverage a small amount of annotated data (supervised deep learning) and a large amount of readily available unannotated data (unsupervised learning), yielding a new data-efficient and accurate semi-supervised semantic segmentation method, all embodied into a single end-to-end trainable convolutional neural networks architecture. Our method is evaluated on representative data from a sandeel survey in the North Sea conducted by the Norwegian Institute of Marine Research. The rigorous experiments validate that our method achieves comparable results utilizing only 40 percent of the annotated data on which the supervised method is trained, by leveraging unannotated data. The code is available at https://github.com/SFI-Visual-Intelligence/PredKlus-semisup-segmentation.

The fully supervised approaches achieve good performance provided that high-quality training data and an appropriate choice for the prediction model are assured. However, it is highly challenging for the echosounder data to obtain the class annotation for each backscattering intensity pixel because this relies on the manual annotation process, which is expensive and error-prone. As a key step in this direction, we propose a novel deep semi-supervised semantic segmentation method that efficient- ly uses a small amount of manually annotated data by com- bining it with a large amount of readily available unannotated data in the learning process. Poster presentation at VI days 2023. The original work is published in IEEE Journal of Oceanic Engineering DOI: 10.1109/JOE.2022.3226214.

The age determination of fish is fundamental to marine resource management. This task is commonly done by analysis of otoliths performed manually by human experts. Otolith images from Greenland halibut acquired by the Institute of Marine Research (Norway) were recently used to train a convolutional neural network (CNN) for automatically predicting fish age, opening the way for requiring less human effort and availability of expertise by means of deep learning (DL). In this study, we demonstrate that applying a CNN model trained on images from one lab (in Norway) does not lead to a suitable performance when predicting fish ages from otolith images from another lab (in Iceland) for the same species. This is due to a problem known as dataset shift, where the source data, i.e., the dataset the model was trained on have different characteristics from the dataset at test stage, here denoted as target data. We further demonstrate that we can handle this problem by using domain adaptation, such that an existing model trained in the source domain is adapted to perform well in the target domain, without requiring extra annotation effort. We investigate four different approaches: (i) simple adaptation via image standardization, (ii) adversarial generative adaptation, (iii) adversarial discriminative adaptation and (iv) self-supervised adaptation. The results show that the performance varies substantially between the methods, with adversarial discriminative and self-supervised adaptations being the best approaches. Without using a domain adaptation approach, the root mean squared error (RMSE) and coefficient of variation (CV) on the Icelandic dataset are as high as 5.12 years and 28.6%, respectively, whereas by using the self-supervised domain adaptation, the RMSE and CV are reduced to 1.94 years and 11.1%. We conclude that careful consideration must be given before DL-based predictors are applied to perform large scale inference. Despite that, domain adaptation is a promising solution for handling problems of dataset shift across image labs.

Every year, marine scientists around the world read thousands of otolith or scale images to determine the age structure of commercial fish stocks. This knowledge is important for fisheries and conservation management. However, the age-reading procedure is time-consuming and costly to perform due to the specialized expertise and labor needed to identify annual growth zones in otoliths. Effective automated systems are needed to increase throughput and reduce cost. DeepOtolith is an open-source artificial intelligence (AI) platform that addresses this issue by providing a web system with a simple interface that automatically estimates fish age by combining otolith images with convolutional neural networks (CNNs), a class of deep neural networks that has been a dominant method in computer vision tasks. Users can upload otolith image data for selective fish species, and the platform returns age estimates. The estimates of multiple images can be exported to conduct conclusions or further age-related research. DeepOtolith currently contains classifiers/regressors for three fish species; however, more species will be included as related work on ageing will be tested and published soon. Herein, the architecture and functionality of the platform are presented. Current limitations and future directions are also discussed. Overall, DeepOtolith should be considered as the first step towards building a community of marine ecologists, machine learning experts, and stakeholders that will collaborate to support the conservation of fishery resources.

With recent abundant availability of high resolution multi-sensor UAV data and rapid development of deep learning models, efficient automatic mapping using deep neural network is becoming a common approach. However, with the ever-expanding inventories of both data and deep neural network models, it can be confusing to know how to choose. Most models expect input as conventional RGB format, but that can be extended to incorporate multi-sensor data. In this study, we re-implement and modify three deep neural network models of various complexities, namely UNET, DeepLabv3+ and Dense Dilated Convolutions Merging Network to use both RGB and near infrared (NIR) data from a multi-sensor UAV dataset over a Norwegian coastal area. The dataset has been carefully annotated by marine experts for coastal habitats. We find that the NIR channel increases UNET performance significantly but has mixed effects on DeepLabv3+ and DDCM. The latter two are capable of achieving best performance with RGB-only. The class-wise evaluation shows that the NIR channel greatly increases the performance in UNET for green, red algae, vegetation and rock. However, the purpose of the study is not to merely compare the models or to achieve the best performance, but to gain more insights on the compatibility between various models and data types. And as there is an ongoing effort in acquiring and annotating more data, we aim to include them in the coming year.

We propose and investigate a method for creating large scale forest height maps at 10 m resolution from Sentinel-2 data using deep neural networks. In addition, we demonstrate how clear-cutting events can be detected in a time series of the resulting forest height maps. The network architecture is a convolutional neural network based on the U-Net architecture. The 13 Sentinel-2 spectral bands are resampled to 10 m spatial resolution and input to the U-Net, which outputs a map with per-pixel forest height estimates. The network is trained with ground truth data acquired from airborne lidar scanning surveys from three different geographical regions. They cover different types of forests: lowland tropical rainforest in the Democratic Republic of Congo, Miombo woodlands (dry forest) in Liwale, Tanzania, and submontane tropical rainforest in Amani, Tanzania. We demonstrate that the trained network generalizes to new geographical regions within the African continent with a mean average error of 4.6 m. This is on-par with a previously published method’s ability to generalize to new geographical regions within the same country. Clear-cutting events are detected using a t-test. The null-hypothesis of the t-test is that the forest height has not changed after any given point in time in the forest height time-series.

Multi-modality data is becoming readily available in remote sensing (RS) and can provide complementary information about the Earth’s surface. Effective fusion of multi-modal information is thus important for various applications in RS, but also very challenging due to large domain differences, noise, and redundancies. There is a lack of effective and scalable fusion techniques for bridging multiple modality encoders and fully exploiting complementary information. To this end, we propose a new multi-modality network (MultiModNet) for land cover mapping of multi-modal remote sensing data based on a novel pyramid attention fusion (PAF) module and a gated fusion unit (GFU). The PAF module is designed to efficiently obtain rich fine-grained contextual representations from each modality with a built-in cross-level and cross-view attention fusion mechanism, and the GFU module utilizes a novel gating mechanism for early merging of features, thereby diminishing hidden redundancies and noise. This enables supplementary modalities to effectively extract the most valuable and complementary information for late feature fusion. Extensive experiments on two representative RS benchmark datasets demonstrate the effectiveness, robustness, and superiority of the MultiModNet for multi-modal land cover classification.

This paper presents a new method for semi-automatic detection of nature interventions inSentinel-2 satellite images with 10 m spatial res-olution. The Norwegian Environment Agency ismaintaining a map of undisturbed nature in Nor-way. U-Net was used for automated detection ofnew roads, as these are often the cause wheneverthe area of undisturbed nature is reduced. Themethod was able to detect many new roads, butwith some false positives and possibly some falsenegatives (i.e., missing new roads). In conclusion,we have demonstrated that automated detection ofnew roads, for the purpose of updating the mapof undisturbed nature, is possible. We have alsosuggested several improvements of the method toimprove its usefulness.

Acoustic target classification in multi-frequency echosounder data is a major interest for the marine ecosystem and fishery management since it can potentially estimate the abundance or biomass of the species. A key problem of current methods is the heavy dependence on the manual categorization of data samples. As a solution, we propose a novel semi-supervised deep learning method leveraging a few annotated data samples together with vast amounts of unannotated data samples, all in a single model. Specifically, two inter-connected objectives, namely, a clustering objective and a classification objective, optimize one shared convolutional neural network in an alternating manner. The clustering objective exploits the underlying structure of all data, both annotated and unannotated; the classification objective enforces a certain consistency to given classes using the few annotated data samples. We evaluate our classification method using echosounder data from the sandeel case study in the North Sea. In the semi-supervised setting with only a tenth of the training data annotated, our method achieves 67.6% accuracy, outperforming a conventional semi-supervised method by 7.0 percentage points. When applying the proposed method in a fully supervised setup, we achieve 74.7% accuracy, surpassing the standard supervised deep learning method by 4.7 percentage points.

Capturing global contextual representations in remote sensing images by exploiting long-range pixel-pixel dependencies has been shown to improve segmentation performance. However, how to do this efficiently is an open question as current approaches of utilising attention schemes, or very deep models to increase the field of view, increases complexity and memory consumption. Inspired by recent work on graph neural networks, we propose the Self-Constructing Graph (SCG) module that learns a long-range dependency graph directly from the image data and uses it to capture global contextual information efficiently to improve semantic segmentation. The SCG module provides a high degree of flexibility for constructing segmentation networks that seamlessly make use of the benefits of variants of graph neural networks (GNN) and convolutional neural networks (CNN). Our SCG-GCN model, a variant of SCG-Net built upon graph convolutional networks (GCN), performs semantic segmentation in an end-to-end manner with competitive performance on the publicly available ISPRS Potsdam and Vaihingen datasets, achieving a mean F1-scores of 92.0% and 89.8%, respectively. We conclude that the SCG-Net is an attractive architecture for semantic segmentation of remote sensing images since it achieves competitive performance with much fewer parameters and lower computational cost compared to related models based on convolutional neural networks.

The aim of the study was to investigate whether context information related to acoustic data resolution could improve automatic acoustic classification of species using echosounder observations.

In this essay, we review some recent advances in developing machine learning (ML) methods for marine science applications in Norway. We focus mostly on deep learning (DL) methods and review the challenges we have faced in the process, including data preparation, (lack of) labelled training data, and interpretability. We also present the partnerships that have been formed between e-science institutions and marine science institutions in Norway. These partnerships have been instrumental in moving this effort forward and have been fuelled by grants from the Norwegian Research Council. The last addition to this collaboration is the recent centres for research-based innovation in Marine Acoustic Abundance Estimation and Backscatter Classification (CRIMAC) and Visual Intelligence (VI).

During the presentation, we investigated how a model trained for automatic fish age prediction on otolith images from one lab could generalize to novel images from another lab. We identified Unsupervised Domain Adaptation based on coGAN (Liu and Tuzel, 2016) and Generate to Adapt (Sankaranarayanan et al., 2018) frameworks as promising methods.

Traffic estimation from very-high-resolution remote-sensing imagery has received increasing interest during the last few years. In this article, we propose an automatic system for estimation of the annual average daily traffic (AADT) using very-high-resolution optical remote-sensing imagery of urban areas in combination with high-quality, but very spatially limited, ground-based measurements. The main part of the system is the vehicle detection, which is based on the deep learning object detection architecture mask region-based convolutional neural network (Mask R-CNN), modified with an image normalization strategy to make it more robust for test images of various conditions and the use of a precise road mask to assist the filtering of driving vehicles from parked ones. Furthermore, to include the high-quality ground-based measurements and to make the traffic estimates more consistent across neighbouring road links, we propose a graph smoothing strategy that utilizes the road network. The fully automatic processing chain has been validated on a set of aerial images covering the city of Narvik, Norway. The precision and recall rate of detecting driving vehicles was 0.74 and 0.66, respectively, and the AADT was estimated with a root mean squared error (RMSE) of 2279 and bias of −383. We conclude that separating driving vehicles from parked ones may be challenging if vehicles are parked along the roads and that for urban environment with short road links several remote-sensing images covering the road links at different time instances are necessary in order to benefit from the remote-sensing images.

Graph Neural Networks (GNNs) have received increasing attention in many fields. However, due to the lack of prior graphs, their use for semantic labeling has been limited. Here, we propose a novel architecture called the Self-Constructing Graph (SCG), which makes use of learnable latent variables to generate embeddings and to self-construct the underlying graphs directly from the input features without relying on manually built prior knowledge graphs. SCG can automatically obtain optimized non-local context graphs from complex-shaped objects in aerial imagery. We optimize SCG via an adaptive diagonal enhancement method and a variational lower bound that consists of a customized graph reconstruction term and a Kullback-Leibler divergence regularization term. We demonstrate the effectiveness and flexibility of the proposed SCG on the publicly available ISPRS Vaihingen dataset and our model SCG-Net achieves competitive results in terms of F1-score with much fewer parameters and at a lower computational cost compared to related pure-CNN based work.

We propose a novel architecture called the Multi-view Self-Constructing Graph Convolutional Networks (MSCG-Net) for semantic segmentation. Building on the recently proposed Self-Constructing Graph (SCG) module, which makes use of learnable latent variables to self-construct the underlying graphs directly from the input features without relying on manually built prior knowledge graphs, we leverage multiple views in order to explicitly exploit the rotational invariance in airborne images. We further develop an adaptive class weighting loss to address the class imbalance. We demonstrate the effectiveness and flexibility of the proposed method on the Agriculture-Vision challenge dataset and our model achieves very competitive results (0.547 mIoU) with much fewer parameters and at a lower computational cost compared to related pure-CNN based work.

Acoustic target classification is the process of assigning observed acoustic backscattering intensity to an acoustic category. A deep learning strategy for acoustic target classification using a convolutional network is developed, consisting of an encoder and a decoder, which allow the network to use pixel information and more abstract features. The network can learn features directly from data, and the learned feature space may include both frequency response and school morphology. We tested the method on multifrequency data collected between 2007 and 2018 during the Norwegian sandeel survey. The network was able to distinguish between sandeel schools, schools of other species, and background pixels (including seabed) in new survey data with an F1 score of 0.87 when tested against manually labelled schools. The network separated schools of sandeel and schools of other species with an F1 score of 0.94. A traditional school classification algorithm obtained substantially lower F1 scores (0.77 and 0.82) when tested against the manually labelled schools. To train the network, it was necessary to develop sampling and preprocessing strategies to account for unbalanced classes, inaccurate annotations, and biases in the training data. This is a step towards a method to be applied across a range of acoustic trawl surveys.

Age-reading of fish otoliths (ear stones) is important for the sustainable management of fish resources. However, the procedure is challenging and requires experienced readers to carefully examine annual growth zones. In a recent study, convolutional neural networks (CNNs) have been demonstrated to perform reasonably well on automatically predicting fish age from otolith images. In the present study, we carefully investigate the prediction rule learned by such neural networks to provide insight into the features that identify certain fish age ranges. For this purpose, a recent technique for visualizing and analyzing the predictions of the neural networks was applied to different versions of the otolith images. The results indicate that supplementary knowledge about the internal structure improves the results for the youngest age groups, compared to using only the contour shape attribute of the otolith. However, the contour shape and size attributes are, in general, sufficient for older age groups. In addition, within specific age ranges we find that the network tends to focus on particular areas of the otoliths and that the most discriminating factors seem to be related to the central part and the outer edge of the otolith. Explaining age predictions from otolith images as done in this study will hopefully help build confidence in the potential of deep learning algorithms for automatic age prediction, as well as improve the quality of the age estimation.

Oceans constitute over 70% of the earth’s surface, and the marine environment and ecosystems are central to many global challenges. Not only are the oceans an important source of food and other resources, but they also play a important roles in the earth’s climate and provide crucial ecosystem services. To monitor the environment and ensure sustainable exploitation of marine resources, extensive data collection and analysis efforts form the backbone of management programmes on global, regional, or national levels. Technological advances in sensor technology, autonomous platforms, and information and communications technology now allow marine scientists to collect data in larger volumes than ever before. But our capacity for data analysis has not progressed comparably, and the growing discrepancy is becoming a major bottleneck for effective use of the available data, as well as an obstacle to scaling up data collection further. Recent years have seen rapid advances in the fields of artificial intelligence and machine learning, and in particular, so-called deep learning systems are now able to solve complex tasks that previously required human expertise. This technology is directly applicable to many important data analysis problems and it will provide tools that are needed to solve many complex challenges in marine science and resource management. Here we give a brief review of recent developments in deep learning, and highlight the many opportunities and challenges for effective adoption of this technology across the marine sciences.