Anders Ueland Waldeland

The Norwegian Computing Center has been working on automating the inspection and monitoring of railway infrastructure for the last 5 years in collaboration with Bane NOR. Together, NR and Bane NOR are developing a mobile and cost-efficient camera system that can be mounted on the front of maintenance trains to record video of the infrastructure. We have been using deep learning–based computer vision on this data to detect faults, identify anomalies, and perform positioning for change detection. In this presentation, Anders Waldeland talks about this work and how we have been using various foundation models and self-supervised learning to advance automatic inspection.

Foundation models (FMs) are transforming the field of Artificial Intelligence (AI) by learning inherent information from vast amounts of unlabeled data, enabling adaptation to numerous applications. Their integration into the Earth Observation (EO) ecosystem promises to revolutionize the information value chain, impacting industry, research, and science. However, EO applications present unique challenges, including the diverse needs for detail or rapid processing, and the variety of data value and sensor characteristics. Models must handle data from multiple sensors at varying ground sampling distances (GSD). Vision Transformers (ViT), often trained using self-supervised learning (SSL), form the backbone of many modern FMs by learning from complex data patterns without explicit supervision. We introduce FM4CS, a versatile foundation model specifically designed for climate and society EO applications. FM4CS aims to address the aforementioned challenges by supporting four different Sentinel sensors: Sentinel-1 SAR, Sentinel-2 MSI, Sentinel-3 OLCI, and Sentinel-3 SLSTR. Inspired by approaches like USat for multi-sensor data handling at native resolutions and FlexiViT for operating across a wide range of patch sizes without retraining, FM4CS employs a single ViT architecture. The model utilizes individual patch embedding layers for each sensor channel, allowing flexibility in processing subsets of spectral bands. It adapts the number of patches per band based on GSD and uses spectral group pooling to manage token sequence length. To accommodate flexible patch sizes, FM4CS incorporates a training procedure where patch size is randomized, allowing the model to adapt dynamically at inference time. For handling the need for positional information of the patches across different sensors, resolutions and patch sizes, FM4CS adapts the 2D ALiBi (Attention by Linear Bias) relative positional encoding scheme. The pre-training dataset for FM4CS is curated to ensure diversity. Instead of stacking small image crops, data is sampled using the Sentinel-2 tiling grid, co-locating Sentinel-1, Sentinel-2, and Sentinel-3 imagery for given locations and time intervals. A stratified sampling approach, based on k-means clustering of ESA WorldCover maps and Sentinel-2 RGB composites, is used to capture the diversity of global land cover and address imbalances. Oceanic data sampling incorporates shipping traffic density, oil and gas installations, and areas with higher probabilities of sea ice and icebergs. The dataset also includes ERA5-Land variables to facilitate multi-modal pretext tasks, leveraging daily statistics for variables such as soil moisture, temperature, and snow cover. FM4CS is trained using several SSL tasks. These include pixel-level input band reconstruction, similar to masked image modeling, where a lightweight ViT decoder reconstructs masked tokens. Additionally, the model predicts existing maps such as ESA WorldCover from Sentinel-1/2 data, and other land cover maps (ESA GlobCover, MOD12Q1) from the Sentinel-3 sensors using a cross-entropy loss. Image-level tasks involve the prediction of ERA5 variables, latitude, longitude and data acquisition month.

To leverage Earth observation (EO) data for large scale analysis, automatic methods is a prerequisite. Since 2012, deep learning (DL) models have brought about a revolutionary change in the analysis of image data and are currently considered state-of-the-art for a broad spectrum of EO tasks. However, a bottleneck with supervised DL models is that they often require a vast amount of labelled data to be trained, and the research community has therefore started to explore alternatives to supervised learning. During the last years, foundation models (FM) signify a change of thinking in computer vision. FMs are trained on a vast volume of unlabeled data and can identify complex patterns due to their large-scale learning capabilities. Typically, an additional head or decoder (small network) is added to the FM, which is trained and adapted to various use-cases by means of a small amount of labelled data. FM have also started to be explored for EO applications, however, current EO-based FMs are limited in terms of handling different modalities with large differences in resolution. Modern FMs are often based on transformers and are trained using self-supervised learning (SSL). There are several SSL schemes in place, including masked autoencoders (MAE) where we mask part of the input data and learn the model to predict the masked data. This is not useful by itself, but the model learns compressed representation of the data, which can be leveraged in downstream applications. This potentially makes the FM more useful than models trained on a limited set of labeled data. The Norwegian Computing Center and UiT – The Arctic University of Norway are in collaboration with user partners Romanian National Meteorological Administration, Danish Meteorological Institute, Polar View and Norwegian Water Resources and Energy Directorate developing a multi-modal FM. The FM is designed to process data from the satellites Sentinel-1 SAR, Sentinel-2 and Sentinel-3 OLCI and SLSTR. The FM is based on vison transformers (ViT) but utilizing the same principle as the USat approach to handle the different resolutions between the modalities. The training of the FM is based on the MAE approach, and to ensure that the SSL work efficiently, we have developed a smart sampling scheme during that provides relevant and diverse training data. In addition to SSL, we have also created a learning task in regressing climate variables from the ERA5 dataset. To train the FM, over 20 TB of Sentinel data was collected and processed using the LUMI supercomputer. The multi-modal FM is demonstrated on the following use-cases: mapping of snow, flood zone mapping, mapping and monitoring of sea ice, iceberg detection, early draught warning and mapping of wetlands. The resolution of the target use-case products are vastly different, e.g. for snow mapping we aim for a ground sampling distance (GSD) of 250m whereas for flood zone mapping we aim for a GSD of 10m. We have therefore trained two versions of the FM: one aiming for high-resolution products with GSD between 10 – 60m, and one aiming for low-resolutions products with GSD above 100m. The downstream tasks are implemented using the open-source framework TerraTorch, which is a flexible fine-tuning framework for geospatial FMs. TerraTorch supports common fine-tuning tasks such as image segmentation and pixel-wise regression along with a selection of task-specific decoder heads.

To leverage Earth observation (EO) data for large scale analysis, automatic methods is a prerequisite. Since 2012, deep learning (DL) models have brought about a revolutionary change in the analysis of image data and are currently considered state-of-the-art for a broad spectrum of EO tasks. However, a bottleneck with supervised DL models is that they often require a vast amount of labelled data to be trained, and the research community has therefore started to explore alternatives to supervised learning. Inspired by the progress in large language models, foundation models (FM) are now being applied extensively in computer vision. FMs are trained on a vast volume of unlabeled data and can identify complex patterns due to their large-scale learning capabilities. Typically, an additional head or decoder (small network) is added to the FM, which is trained and adapted to various use-cases by means of a small amount of labelled data. FM have also started to be explored for EO applications, however, current EO-based FMs are limited in terms of handling different modalities with large differences in resolution. Modern FMs are often based on transformers and are trained using self-supervised learning (SSL). There are several SSL schemes in place, including masked autoencoders (MAE) where we mask part of the input data and learn the model to predict the masked data. This is not useful by itself, but the model learns compressed representation of the data, which can be leveraged in downstream applications. This potentially makes the FM more useful than models trained on a limited set of labeled data. The Norwegian Computing Center and UiT – The Arctic University of Norway are in collaboration with user partners Romanian National Meteorological Administration, Danish Meteorological Institute, Polar View and Norwegian Water Resources and Energy Directorate developing a multi-modal FM. The FM is designed to process data from the satellites Sentinel-1 SAR, Sentinel-2 and Sentinel-3 OLCI and SLSTR. The FM is based on vison transformers (ViT) but utilizing the same principle as the USat approach to handle the different resolutions between the modalities. The training of the FM is based on the MAE approach, and to ensure that the SSL work efficiently, we have developed a smart sampling scheme during that provides relevant and diverse training data. In addition to SSL, we have also created a learning task in regressing climate variables from the ERA5 dataset. To train the FM, over 20 TB of Sentinel data was collected and processed using the LUMI supercomputer. The multi-modal FM is demonstrated on the following use-cases: mapping of snow, flood zone mapping, mapping and monitoring of sea ice, iceberg detection, early draught warning and mapping of wetlands. The downstream tasks are implemented using the open-source framework TerraTorch, which is a flexible fine-tuning framework for geospatial FMs. The FM4CS model is one of the first to apply Sentinel-3 data, which makes it attractive for climate applications.

Accurate uncertainty information associated with essential climate variables (ECVs) is crucial for reliable climate modeling and understanding the spatiotemporal evolution of the Earth system. In recent years, geoscience and climate scientists have benefited from rapid progress in deep learning to advance the estimation of ECV products with improved accuracy. However, the quantification of uncertainties associated with the output of such deep learning models has yet to be thoroughly adopted. This survey explores the types of uncertainties associated with ECVs estimated from deep learning and the techniques to quantify them. The focus is on highlighting the importance of quantifying uncertainties inherent in ECV estimates, considering the dynamic and multifaceted nature of climate data. The survey starts by clarifying the definition of aleatoric and epistemic uncertainties and their roles in a typical satellite observation processing workflow, followed by bridging the gap between conventional statistical and deep learning views on uncertainties. Then, we comprehensively review the existing techniques for quantifying uncertainties associated with deep learning algorithms, focusing on their application in ECV studies. The specific need for modification to fit the requirements from both the Earth observation side and the deep learning side in such interdisciplinary tasks is discussed. Finally, we demonstrate our findings with two ECV examples, snow cover and terrestrial water storage, and provide our perspectives for future research.

We propose and investigate a method for creating large scale forest height maps at 10 m resolution from Sentinel-2 data using deep neural networks. In addition, we demonstrate how clear-cutting events can be detected in a time series of the resulting forest height maps. The network architecture is a convolutional neural network based on the U-Net architecture. The 13 Sentinel-2 spectral bands are resampled to 10 m spatial resolution and input to the U-Net, which outputs a map with per-pixel forest height estimates. The network is trained with ground truth data acquired from airborne lidar scanning surveys from three different geographical regions. They cover different types of forests: lowland tropical rainforest in the Democratic Republic of Congo, Miombo woodlands (dry forest) in Liwale, Tanzania, and submontane tropical rainforest in Amani, Tanzania. We demonstrate that the trained network generalizes to new geographical regions within the African continent with a mean average error of 4.6 m. This is on-par with a previously published method’s ability to generalize to new geographical regions within the same country. Clear-cutting events are detected using a t-test. The null-hypothesis of the t-test is that the forest height has not changed after any given point in time in the forest height time-series.

Acoustic target classification is the process of assigning observed acoustic backscattering intensity to an acoustic category. A deep learning strategy for acoustic target classification using a convolutional network is developed, consisting of an encoder and a decoder, which allow the network to use pixel information and more abstract features. The network can learn features directly from data, and the learned feature space may include both frequency response and school morphology. We tested the method on multifrequency data collected between 2007 and 2018 during the Norwegian sandeel survey. The network was able to distinguish between sandeel schools, schools of other species, and background pixels (including seabed) in new survey data with an F1 score of 0.87 when tested against manually labelled schools. The network separated schools of sandeel and schools of other species with an F1 score of 0.94. A traditional school classification algorithm obtained substantially lower F1 scores (0.77 and 0.82) when tested against the manually labelled schools. To train the network, it was necessary to develop sampling and preprocessing strategies to account for unbalanced classes, inaccurate annotations, and biases in the training data. This is a step towards a method to be applied across a range of acoustic trawl surveys.

The deep learning revolution in computer vision has enabled a potential for creating new value chains for Earth observation that significantly enhances the analysis of satellite data for tasks like land cover mapping, change analysis, and object detection. We demonstrate a deep learning based value chain for the task of mapping vegetation height in the Liwale region in Tanzania using Sentinel-1 and −2 data. As ground truth data we use lidar measurements, which are processed to provide the average vegetation height per Sentinel-2 pixel grid (10 m). We apply the UNet, which is a widely used neural network for segmentation tasks in computer vision, to estimate average vegetation height from the Sentinel data. Preliminary results show that we are able to map the forest extent with high accuracy, with an RMSE of 3.5 m for Sentinel-2 data and 4.6 m for the Sentinel-1 data.

This project seeks to develop heavily automated analysis of digital topographic data to extract archaeological information and to expedite the creation of national-scaled mapping. Drawing on developments in computer vision this has the potential to fundamentally recast the capacity of archaeological prospection and survey to cover large areas and deal with mass data, breaking a dependency on human resource. Without such developments the potential of the vast amount of archaeological information embedded in large topographic and image-based datasets cannot be realised to inform our knowledge and understanding of Scotland’s Historic Environment. A heavily automated computational approach and the increasing availability of large datasets put the creation of systematic national-scaled archaeological mapping of Scotland within reach. The purpose of this proof of concept project was to run an assessment of existing developments in a Norwegian case study against digital topographic data for Arran, providing outputs that may be assessed for their applicability at a national scale.

This article presents results of a case study within a project that seeks to develop heavily automated analysis of digital topographic data to extract archaeological information and to expedite large area mapping. Drawing on developments in computer vision and machine learning, this has the potential to fundamentally recast the capacity of archaeological prospection to cover large areas and deal with mass data, breaking a dependency on human resource. Without such developments, the potential of the vast amount of archaeological information embedded in large topographic and image‐based datasets cannot be realized. The purpose of the case study reported on here is to assess existing developments in a Norwegian study against digital topographic data for the island of Arran, Scotland, examining the transferability of the approach and providing a proof of concept in a Scottish context. For Arran, three monument classes were assessed – prehistoric roundhouses, shieling huts of medieval or post‐medieval date, and small clearance cairns. These present different challenges to detection, with preliminary results ranging from a manageable mix of false positives and true identifications to the chaotic. The influence of variable morphology and the occurrence of other, largely natural, objects of confusion in the landscape is discussed, highlighting the potential improvements in automated detection routines offered by adding anthropogenic and natural false positives to additional confusion classes.

How and why can Deep Learning be used for seismic interpretation? The machine learning technique called deep learning is revolutionizing the field of computer vision. A central part of deep learning is convolutional neural networks (CNN). This E-lecture gives a simple and intuitive introduction to CNNs in the context of seismic interpretation.