Mark Anderson

Text sanitization is the task of redacting a document to mask all occurrences of (direct or indirect) personal identifiers, with the goal of concealing the identity of the individual(s) referred in it. In this paper, we consider a two-step approach to text sanitization and provide a detailed analysis of its empirical performance on two recently published datasets: the Text Anonymization Benchmark (Pilán et al., 2022) and a collection of Wikipedia biographies (Papadopoulou et al., 2022a). The text sanitization process starts with a privacy-oriented entity recognizer that seeks to determine the text spans expressing identifiable personal information. This privacy-oriented entity recognizer is trained by combining a standard named entity recognition model with a gazetteer populated by person-related terms extracted from Wikidata. The second step of the text sanitization process consists in assessing the privacy risk associated with each detected text span, either isolated or in combination with other text spans. We present five distinct indicators of the re-identification risk, respectively based on language model probabilities, text span classification, sequence labelling, perturbations, and web search. We provide a contrastive analysis of each privacy indicator and highlight their benefits and limitations, notably in relation to the available labeled data.

This report describes the research related to use case 3 within pilot ♯6 of the FAME project: Embedding Climatic Predictions in Property Insurance Products. The changing climate poses risks to the global economy in various ways. In addition to the immediate impacts on the real estate sector, e.g., due to changing living conditions, and impacts on the insurance sector as a result of more frequent and intense weather-related disasters, it calls for the transition to a greener and more sustainable economy, which comes with a significant price tag. While the latter risk cannot be directly related to specific weather events, an increasing body of research suggests that climate risk indices constructed through textual analysis of newspaper articles are able to represent the different types of risks and can thus help build climate risk hedge portfolios. In this report, we document our own efforts to construct such a news-based climate risk index and analyze whether climate-focused funds perform differently than a benchmark representing the general market. Our risk index is constructed from an analysis of climate change-related newspaper articles in the New York Times which were further filtered by utilizing large language model (LLM) with zero-shot classification. Both full and subcategory-based indices are defined at a daily resolution and were aggregated to both weekly and monthly resolution in order to analyze impacts on stock market returns at different time scales. Our analysis, however, does not show a clear and robust link between the climate risk index and the stock market returns of climate-focused funds compared to the general market. A more refined selection of ’green’ vs. ’brown’ stocks may be required to see significant, climate risk index-dependent differences in performance that can be exploited for the purpose of portfolio optimization.

Statistisk sentralbyrå (SSB) publiserer månedlig statistikk over utenrikshandel med varer (UHV). Formålet med statistikken er å gi en oversikt over varestrømmene mellom Norge og utlandet. Eksport og import av varer er sentrale størrelser i det samfunnsøkonomiske bildet, og følgelig er det viktig at statistikken som publiseres er presis. Norsk Regnesentral (NR) har analysert tolldata på import av varer fra TollVesenets INformasjonssystem med Næringslivet (TVINN). Alle importerte varer registreres gjennom en varelinje, med et varenummer som beskriver varetypen fra Tolltariffen, og feil varenummer er en vanlig feiltype. I en del tilfeller innebærer en slik feilregistrering manglende samsvar mellom varenummer og varebeskrivelsen. I dette prosjektet har vi brukt en språkmodell til å innkode varebeskrivelsen, et fritekstfelt, for så å predikere varenummeret til en varelinje ved hjelp av logistisk regresjon. I tillegg har vi eksperimentert med å bruke gradientforsterkede tremodeller til samme formål, basert på andre felt fra hver varelinje. Disse modellene er trent på varelinjer fra enten 2017 - 2021, eller 2020 - 2023, som er kontrollert og har fått varenummeret sitt korrigert. Ved å teste modellene på uavhengige testdata fra 2024, finner vi at modellene tilpasset varebeskrivelsene oppnår en rimelig høy treffsikkerhet, med en treffsikkerhet på ca. 65% for tekstmodellen tilpasset varelinjer fra perioden 2020 - 2023, evaluert på litt over 24 tusen korrigerte varelinjer. De gradientforsterkede tremodellene har ikke oppnådd tilsvarende ytelse, med en maksimal treffsikkerhet på ca. 11%, antageligvis delvis grunnet overtilpasning til treningsdataene.

Data privacy is an important facet of modern life. It is especially important when considering data that carries potentially sensitive information such as in medical or legal documents. However, it is particularly difficult to ensure private information has been removed or masked in unstructured data, e.g. free-flowing text. This is a major concern in the current large language model paradigm in natural language processing where massive amounts of data is processed by these models without any level of scrutiny as to what the data actually contains. To this end, many practitioners have attempted to automatically detect and remove personal identifiable information (PII) from documents using a variety of different methods. Most approaches used to detect PII are based on methods that require labelled data to train a machine learning model to classify words or spans of words as PII. Recently a new method captured the disclosure risk by characterizing the semantic relationships between entities in a document based on word embeddings, without the requirement of manual tagging data [1]. There are two main concerns associated with the performance of these methods. The first is to what degree a given method ensures privacy protection. The second is what level of utility is preserved in the document after the PII have been removed. The most straightforward and standard metrics used to evaluate these methods are recall for privacy protection and precision for utility preservation. However, such empirical evaluation can be misleading. We evaluate the method in [1] as a case study and show that these evaluation metrics fail to give us a complete picture of the model’s behavior and performance. We present an analysis showing why the proposed method is not as effective as it seems when using these metrics for evaluation and discuss how better to use them. We also discuss how to improve evaluation by using other metrics, building on the work presented in [2]. Reference Style [1] Fadi Hassan, David Sánchez, and Josep Domingo-Ferrer (2023) “Utility-Preserving Privacy Protection of Textual Documents via Word Embeddings” IEEE Transactions on Knowledge and Data Engineering Volume 35, Number 1: page 1058 – page 1071. [2] Ildikó Pilán, Pierre Lison, Lilja Øvrelid, Anthi Papadopoulou, David Sánchez, and Montserrat Batet (2022) “The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymization” Computational Linguistics Volume 28, Issue 4: page 1053 – page 1101.

Memory is fleeting. To avoid information loss, humans need to recode verbal stimuli into chunks of limited duration, each containing multiple words. Chunk duration may also be limited neurally by the wavelength of periodic brain activity, so-called neural oscillations. While both cognitive and neural constraints predict some degree of behavioral regularity in processing, this remains to be shown. Our analysis of self-paced reading data from 181 participants reveals periodic patterns at a frequency of 2 Hz. We defined multi-word chunks by using a computational formalization based on dependency annotations and part-of-speech tags. Potential chunk outputs were first generated from the computational formalization and the final chunk outputs were selected based on normalized pointwise mutual information. We show that behavioral periodicity is time-aligned to multi-word chunks, suggesting that the multi-word chunks generated from local dependency clusters may minimize memory demands. This is the first evidence that sentence processing behavior is periodic, consistent with a role of both memory constraints and endogenous electrophysiological rhythms in the formation of chunks during language comprehension.