Sjefsforsker
Pierre Lison
- Avdeling Statistisk modellering og maskinlæring
- Telefonnummer +47 22 85 25 77
- E-post plison@nr.stage.dekodes.no
Vis beskrivelsesinformasjon
Skjul beskrivelsesinformasjon
- Sjefsforsker ved Norsk Regnesentral
- Førsteamanuensis II ved Universitetet i Oslo
OM
Mine viktigste forskningsinteresser ligger innen naturlig språkprosessering (NLP) og maskinlæring, spesielt trening, tilpasning og evaluering av store språkmodeller (LLM-er), samt hvordan disse kan tas i bruk i ulike anvendelser.
I løpet av min forskerkarriere har jeg arbeidet med temaer som talebaserte dialogsystemer, storskala informasjonsuttrekk, personvern i data, nevrale maskinoversettelser og samhandling mellom mennesker og roboter.
Jeg er spesielt opptatt av forskningsspørsmål i skjæringspunktet mellom språkbehandling og andre fagområder – både natur- og samfunnsvitenskapelige. Jeg deltar også i flere forsknings- og utviklingsprosjekter med fokus på innovasjon, hvor vi undersøker hvordan store språkmodeller og maskinlæring kan brukes til å løse praktiske utfordringer i offentlig og privat sektor.
Bakgrunn
Jeg er opprinnelig fra Belgia og ble uteksaminert fra Universitetet i Louvain i 2006 med en grad i informatikk og ingeniørvitenskap. Med økende interesse for koblingen mellom informatikk og språkvitenskap flyttet jeg til Saarbrücken i Tyskland for å ta en mastergrad i språkvitenskap og teknologi. Jeg fullførte graden i 2008 og jobbet deretter som forsker ved det tyske forskningssenteret for kunstig intelligens (DFKI), hvor jeg deltok i flere EU-finansierte prosjekter om utvikling av dialogsystemer for samhandling mellom mennesker og roboter.
I 2011 flyttet jeg til Norge for å ta en doktorgrad i språkgruppa ved Universitetet i Oslo. I 2014 forsvarte jeg doktoravhandlingen min om sannsynlighetsbaserte metoder for dialogstyring, og jobbet deretter i to år som postdoktor i samme gruppe med dialogmodellering for statistisk maskinoversettelse.
I 2016 begynte jeg som forsker ved Norsk Regnesentral, hvor jeg jobber med ulike forsknings- og utviklingsprosjekter innen språkprosessering og maskinlæring. To av mine nyeste prosjekter er CLEANUP, som utviklet datadrevne metoder for å fjerne personopplysninger fra tekstdata, og GraphDial, som handlet om dialogstyring og bruk av kunnskapsgrafer for å representere dialogtilstanden i komplekse samtaledomener. Andre prosjekter jeg har vært involvert i inkluderer SAFERS (taleanalyse for nødetater), DialMT (dialogmodellering for maskinoversettelse), AICOM (språklig analyse av samspill mellom mennesker og store språkmodeller), Oslo Analytics, og nylig CyberRisk (cyber-trusselintelligens og risikostyring).
I tillegg til hovedstillingen som sjefsforsker ved NR har jeg også en bistilling som førsteamanuensis II ved språkgruppa ved Universitetet i Oslo, hvor jeg bidrar i flere kurs innen maskinlæring og naturlig språkprosessering. Jeg har også tidligere vært medlem av Akademiet for yngre forskere.
Prosjekter
- Maskinlæring
- Språkteknologi
Anonymisering av tekst (CLEANUP)
- Maskinlæring
- Språkteknologi
- Digital sikkerhet og personvern
Delautomatisering av digital risikostyring
- Maskinlæring
Hvordan forstår vi maskiner som snakker til oss?
Publikasjoner
- 97 publikasjoner funnet
Manzanares-Salor, Benet; Sánchez, David og Lison, Pierre. (2026).
Unsupervised utility evaluation of text anonymization methods via neural language models.
Lison, Pierre; Ruenes, David Sánchez og Stalla-Bourdillon, Sophie. (2026).
Search Data, Privacy, and the Limits of Heuristics: A Critical Reading of the EC's Preliminary Findings against Alphabet.
NVA
Nettsider (opplysningsmateriale)
Lison, Pierre og Schwemer, Sebastian Felix. (2026).
Ideen om en innholdsavgift for KI brer seg.
NVA
Kronikk
Vis sammendrag
Å kompensere mennesker som bidrar med originalt innhold, handler ikke bare om rettferdighet. Det er også en investering i et bærekraftig, digitalt økosystem.
Papadopoulou, Anthi; Lison, Pierre; Anderson, Mark David; Øvrelid, Lilja og Pilán, Ildikó. (2026).
Neural text sanitization with privacy risk indicators: an empirical analysis.
Vis sammendrag
Text sanitization is the task of redacting a document to mask all occurrences of (direct or indirect) personal identifiers, with the goal of concealing the identity of the individual(s) referred in it. In this paper, we consider a two-step approach to text sanitization and provide a detailed analysis of its empirical performance on two recently published datasets: the Text Anonymization Benchmark (Pilán et al., 2022) and a collection of Wikipedia biographies (Papadopoulou et al., 2022a). The text sanitization process starts with a privacy-oriented entity recognizer that seeks to determine the text spans expressing identifiable personal information. This privacy-oriented entity recognizer is trained by combining a standard named entity recognition model with a gazetteer populated by person-related terms extracted from Wikidata. The second step of the text sanitization process consists in assessing the privacy risk associated with each detected text span, either isolated or in combination with other text spans. We present five distinct indicators of the re-identification risk, respectively based on language model probabilities, text span classification, sequence labelling, perturbations, and web search. We provide a contrastive analysis of each privacy indicator and highlight their benefits and limitations, notably in relation to the available labeled data.
Høst, Anders Mølmen; Lison, Pierre og Moonen, Leon. (2026).
A Systematic Approach to Predict the Impact of Cybersecurity Vulnerabilities Using LLMs.
Vis sammendrag
Vulnerability databases, such as the National Vulnerability Database (NVD), offer detailed descriptions of Common Vulnerabilities and Exposures (CVEs), but often lack information on their real-world impact, such as the tactics, techniques, and procedures (TTPs) that adversaries may use to exploit the vulnerability. However, manually linking CVEs to their corresponding TTPs is a challenging and time-consuming task, and the high volume of new vulnerabilities published annually makes automated support desirable.This paper introduces Triage, a two-pronged automated approach that uses Large Language Models (LLMs) to map CVEs to relevant techniques from the Att&ck knowledge base. We first prompt an LLM with instructions based on MITRE’s CVE Mapping Methodology to predict an initial list of techniques. This list is then combined with the results from a second LLM-based module that uses in-context learning to map a CVE to relevant techniques. This hybrid approach strategically combines rule-based reasoning with data-driven inference. Our evaluation reveals that in-context learning outperforms the individual mapping methods, and the hybrid approach improves recall of exploitation techniques. We also find that GPT-4o-mini performs better than Llama3.3-70B on this task. Overall, our results show that LLMs can be used to automatically predict the impact of cybersecurity vulnerabilities and Triage makes the process of mapping CVEs to Att&ck more efficient.
Kennington, Casey; Lison, Pierre og Schlangen, David. (2025).
Prior Lessons of Incremental Dialogue and Robot Action Management for the Age of Language Models.
Vis sammendrag
Efforts towards endowing robots with the ability to speak have benefited from recent advancements in natural language processing, in particular large language models. However, current language models are not fully incremental, as their processing is inherently monotonic and thus lack the ability to revise their interpretations or output in light of newer observations. This monotonicity has important implications for the development of dialogue systems for human–robot interaction. In this paper, we review the literature on interactive systems that operate incrementally (i.e., at the word level or below it). We motivate the need for incremental systems, survey incremental modeling of important aspects of dialogue like speech recognition and language generation. Primary focus is on the part of the system that makes decisions, known as the dialogue manager. We find that there is very little research on incremental dialogue management, offer some requirements for practical incremental dialogue management, and implications of incremental dialogue for embodied, robotic platforms in the age of large language models.
Walker, Nicholas Thomas; Ultes, Stefan og Lison, Pierre. (2025).
Graph-to-Text Approach to Knowledge-Grounded Response Generation in Human–Robot Interaction.
Vis sammendrag
Knowledge graphs are often used to represent structured information in a flexible and efficient manner, but their use in situated dialogue remains under-explored. This paper presents a novel conversational model for human--robot interaction that rests upon a graph-based representation of the dialogue state. The knowledge graph representing the dialogue state is continuously updated with new observations from the robot sensors, including linguistic, situated and multimodal inputs, and is further enriched by other modules, in particular for spatial understanding. The neural conversational model employed to respond to user utterances relies on a simple but effective graph-to-text mechanism that traverses the dialogue state graph and converts the traversals into a natural language form. This conversion of the state graph into text is performed using a set of parameterized functions, and the values for those parameters are optimized based on a small set of Wizard-of-Oz interactions. After this conversion, the text representation of the dialogue state graph is included as part of the prompt of a large language model used to decode the agent response. The proposed approach is empirically evaluated through a user study with a humanoid robot that acts as conversation partner to evaluate the impact of the graph-to-text mechanism on the response generation. After moving a robot along a tour of an indoor environment, participants interacted with the robot using spoken dialogue and evaluated how well the robot was able to answer questions about what the robot observed during the tour. User scores suggest an improvement in the perceived factuality of the robot responses when the graph-to-text approach is employed compared to a baseline using inputs structured as semantic triples.
Baste, Øystein Flø; Cyndecka, Malgorzata Agnieszka; Esayas, Samson Yoseph; Langford, Malcolm; Lison, Pierre og Weitzenboeck, Emily Mary. (2025).
Open Justice Data in Europe: A Patchwork.
Vis sammendrag
The publication of court judgments is essential to upholding rule of law and democratic norms as well as facilitating legal research, and new legal technologies. However, many European states struggled to transition to online publication at scale. In this article we address three questions: what are the obligations of states to publish judgments; which states are making progress; and what are the challenges and solutions in ensuring greater publicity? We examine the overarching duties in the ECHR and EU law and the relevant legal requirements and practice in 12 national jurisdictions and two regional courts. Our findings show tremendous variation in duties and practice, and identify barriers to progress (legal, organisational, and budgetary) but also promising innovative solutions in certain jurisdictions. Ultimately, while this publication diversity provides a form of experimental governance, it would be timely to move towards common standards and approaches.
Helstad, Gina; Lison, Pierre; Bjørnstad-Tuveng, Elin og Nytrøen, Kari. (2025).
Digitising health history: The creation, function and implementation of the Norwegian Health Archives Registry.
Vis sammendrag
Context: The Norwegian Health Archives Registry (NHAR) is a national initiative dedicated to digitising, centralising, and providing access to historical full-text patient health records (PHRs) for research purposes. Established in 2019, NHAR includes PHRs from the deceased population in Norway’s specialist healthcare services, offering a unique long-term data source for future research. NHAR has now digitised 1.7 million paper-based PHRs, covering medical history dating back to 1875. The registry is now expanding to include digital-born PHRs. Aim: This article describes NHAR’s innovation potential as a health registry, its data management processes, and the integration of artificial intelligence (AI) tools to facilitate data management and research in compliance with strict health data regulations. Practice innovation: NHAR’s data value chain includes structured metadata acquisition, large-scale digitisation and secure data delivery for research. The workflow includes a custom optical character recognition (OCR) tool tailored to Norwegian medical terminology, concept-based search tools for unstructured clinical full text and robust strategies for long-term data management. A novel AI-based de-identification system automatically detects and masks personal identifiers in digitised PHRs. Lessons learned: Despite these innovations, challenges persist in processing handwritten and historical PHRs due to OCR limitations and language-specific complexities. Key challenges include improving data quality, enhancing OCR accuracy and refining AI tools for information retrieval, data extraction and de-identification. Conclusion: NHAR offers significant potential for interdisciplinary research across various medical fields. Implications for health information management practice: NHAR establishes a foundation for secure access to historical health data and introduces advanced data management strategies to facilitate future research.
Pilán, Ildikó; Manzanares-Salor, Benet; Sánchez, David og Lison, Pierre. (2025).
Truthful text sanitization guided by inference attacks.
Vis sammendrag
Text sanitization aims to rewrite parts of a document to prevent disclosure of personal information. The central challenge of text sanitization is to strike a balance between privacy protection (avoiding the leakage of personal information) and utility preservation (retaining as much as possible of the document’s original content). To this end, we introduce a novel text sanitization method based on generalizations, that is, broader but still informative terms that subsume the semantic content of the original text spans. The approach relies on the use of instruction-tuned large language models (LLMs) and is divided into two stages. Given a document including text spans expressing personally identifiable information (PII), the LLM is first applied to obtain truth-preserving replacement candidates for each text span and rank them according to their abstraction level. Those candidates are then evaluated for their ability to protect privacy by conducting inference attacks with the LLM. Finally, the system selects the most informative replacement candidate shown to be resistant to those attacks. This two-stage process produces replacements that effectively balance privacy and utility.
We also present novel metrics to evaluate these two aspects without needing to manually annotate documents. Results on the Text Anonymization Benchmark show that the proposed approach, implemented with Mistral 7B Instruct, leads to enhanced utility, with only a marginal (<1 p.p.) increase in re-identification risk compared to fully suppressing the original spans. Furthermore, our approach is shown to be more truth-preserving than existing methods such as Microsoft Presidio’s synthetic replacements.
Kåsene, Vebjørn Haug og Lison, Pierre. (2025).
Following Route Instructions using Large Vision-Language Models: A Comparison between Low-level and Panoramic Action Spaces.
Vis sammendrag
Vision-and-Language Navigation (VLN) refers to the task of enabling autonomous robots to navigate unfamiliar environments by following natural language instructions. While recent Large Vision-Language Models (LVLMs) have shown promise in this task, most current VLM systems rely on models specifically designed and optimized for navigation, leaving the potential of off-the-shelf LVLMs underexplored. Furthermore, while older VLN approaches used low-level action spaces with egocentric views and atomic actions (such as "turn left" or "move forward"), newer models tend to favor panoramic action spaces with discrete navigable viewpoints. This paper investigates (1) whether off-the-shelf LVLMs (fine-tuned without architectural modifications or simulator-based training) can effectively support VLN tasks and (2) whether such models can support both low-level and panoramic action paradigms. To this end, we fine-tune the open-source model Qwen2.5-VL-3B-Instruct on the Room-to-Room (R2R) dataset and evaluate its empirical performance across both low-level and panoramic action spaces. The best resulting model achieves a 41% success rate on the R2R test set, demonstrating that while off-the-shelf LVLMs can learn to perform Vision-and-Language Navigation, they still lag behind models specifically designed for this task.
Walker, Nicholas Thomas; Lison, Pierre; Hilgendorf, Laetitia; Wagner, Nicolas og Ultes, Stefan. (2025).
Retrieving Relevant Knowledge Subgraphs for Task-Oriented Dialogue.
Vis sammendrag
In this paper, we present an approach for extracting knowledge graph information for retrieval augmented generation in dialogue systems. Knowledge graphs are a rich source of background information, but the inclusion of more potentially useful information in a system prompt risks decreased model performance from excess context. We investigate a method of retrieving relevant subgraphs of maximum relevance and minimum size by framing this trade-off as a Prize-collecting Steiner Tree problem. The results of our user study and analysis indicate promising efficacy of a simple subgraph retrieval approach compared with a top-K retrieval model.
Berg, Margareta; Pilán, Ildikó; Falkum, Ingrid Lossius og Lison, Pierre. (2025).
Pragmatic Reasoning for Irony Detection With Large Language Models in English and Norwegian - SEMDIAL.
Vis sammendrag
This study investigates the ‘pragmatic abilities’ of large language models (LLMs) – both standard and reasoning-optimized – across two languages (English and Norwegian). Based on an existing experimental study on children’s irony comprehension, we found that LLMs largely identified irony, but performance was poorer in Norwegian due to translation challenges.
Charpentier, Lucas Georges Gabriel og Lison, Pierre. (2025).
Re-identification of De-identified Documents with Autoregressive Infilling.
Vis sammendrag
Documents revealing sensitive information about individuals must typically be de-identified. This de-identification is often done by masking all mentions of personally identifiable information (PII), thereby making it more difficult to uncover the identity of the person(s) in question. To investigate the robustness of de-identification methods, we present a novel, RAG-inspired approach that attempts the reverse process of re-identification based on a database of documents representing background knowledge. Given a text in which personal identifiers have been masked, the re-identification proceeds in two steps. A retriever first selects from the background knowledge passages deemed relevant for the re-identification. Those passages are then provided to an infilling model which seeks to infer the original content of each text span. This process is repeated until all masked spans are replaced. We evaluate the re-identification on three datasets (Wikipedia biographies, court rulings and clinical notes). Results show that (1) as many as 80% of de-identified text spans can be successfully recovered and (2) the re-identification accuracy increases along with the level of background knowledge.
Kennington, Casey; Lison, Pierre og Schlangen, David. (2025).
Incremental Dialogue Management: Survey, Discussion, and Implications for HRI.
Vis sammendrag
Efforts towards endowing robots with the ability to speak have benefited from recent advancements in NLP, in particular large language models. However, as powerful as current models have become, they still operate on sentence or multi-sentence level input, not on the word-by-word input that humans operate on, affecting the degree of responsiveness that they offer, which is critical in situations where humans interact with robots using speech. In this paper, we review the literature on interactive systems that operate incrementally (i.e., at the word level or below it). We motivate the need for incremental systems, survey incremental modeling of important aspects of dialogue like speech recognition and language generation. Primary focus is on the part of the system that makes decisions, known as the dialogue manager. We find that there is very little research on incremental dialogue management, offer some requirements for practical incremental dialogue management, and the implications of incremental dialogue for embodied, robotic platforms.
Lison, Pierre. (2024).
Automated de-identication of scanned patient records: Evaluation report.
NVA
Rapport
Manzanares-Salor, Benet; Sánchez, David og Lison, Pierre. (2024).
Evaluating the disclosure risk of anonymized documents via a machine learning-based re-identification attack.
Vis sammendrag
The availability of textual data depicting human-centered features and behaviors is crucial for many data mining and machine learning tasks. However, data containing personal information should be anonymized prior making them available for secondary use. A variety of text anonymization methods have been proposed in the last years, which are standardly evaluated by comparing their outputs with human-based anonymizations. The residual disclosure risk is estimated with the recall metric, which quantifies the proportion of manually annotated re-identifying terms successfully detected by the anonymization algorithm. Nevertheless, recall is not a risk metric, which leads to several drawbacks. First, it requires a unique ground truth, and this does not hold for text anonymization, where several masking choices could be equally valid to prevent re-identification. Second, it relies on human judgements, which are inherently subjective and prone to errors. Finally, the recall metric weights terms uniformly, thereby ignoring the fact that the influence on the disclosure risk of some missed terms may be much larger than of others. To overcome these drawbacks, in this paper we propose a novel method to evaluate the disclosure risk of anonymized texts by means of an automated re-identification attack. We formalize the attack as a multi-class classification task and leverage state-of-the-art neural language models to aggregate the data sources that attackers may use to build the classifier. We illustrate the effectiveness of our method by assessing the disclosure risk of several methods for text anonymization under different attack configurations. Empirical results show substantial privacy risks for most existing anonymization methods.
Hassan, Syed Zohaib; Lison, Pierre og Halvorsen, Pål. (2024).
Enhancing Naturalness in LLM-Generated Utterances through Disfluency Insertion.
Vis sammendrag
Disfluencies are a natural feature of spontaneous human speech but are typically absent from the outputs of Large Language Models (LLMs). This absence can diminish the perceived naturalness of synthesized speech, which is an important criteria when building conversational agents that aim to mimick human behaviours. We show how the insertion of disfluencies can alleviate this shortcoming. The proposed approach involves (1) fine-tuning an LLM with Low-Rank Adaptation (LoRA) to incorporate various types of disfluencies into LLM-generated utterances and (2) synthesizing those utterances using a text-to-speech model that supports the generation of speech phenomena such as disfluencies. We evaluated the quality of the generated speech across two metrics: intelligibility and perceived spontaneity. We demonstrate through a user study that the insertion of disfluencies significantly increase the perceived spontaneity of the generated speech. This increase came, however, along with a slight reduction in intelligibility.
Pilán, Ildikó; Prévot, Laurent; Buschmeier, Hendrik og Lison, Pierre. (2024).
Conversational Feedback in Scripted versus Spontaneous Dialogues: A Comparative Analysis.
Vis sammendrag
Scripted dialogues such as movie and TV subtitles constitute a widespread source of training data for conversational NLP models. However, there are notable linguistic differences between these dialogues and spontaneous interactions, especially regarding the occurrence of communicative feedback such as backchannels, acknowledgments, or clarification requests. This paper presents a quantitative analysis of such feedback phenomena in both subtitles and spontaneous conversations. Based on conversational data spanning eight languages and multiple genres, we extract lexical statistics, classifications from a dialogue act tagger, expert annotations and labels derived from a fine-tuned Large Language Model (LLM). Our main empirical findings are that (1) communicative feedback is markedly less frequent in subtitles than in spontaneous dialogues and (2) subtitles contain a higher proportion of negative feedback. We also show that dialogues generated by standard LLMs lie much closer to scripted dialogues than spontaneous interactions in terms of communicative feedback.
Falkum, Ingrid Lossius og Lison, Pierre. (2023).
Er prateroboten ChatGPT en klok samtalepartner eller papegøye?
NVA
Kronikk
Vis sammendrag
https://forskersonen.no/chatgpt-kronikk-kunstig-intelligens/er-prateroboten-chatgpt-en-klok-samtalepartner-eller-papegoye/2157206
Olstad, Annika Willoch; Papadopoulou, Anthi og Lison, Pierre. (2023).
Generation of Replacement Options in Text Sanitization.
NVA
Vitenskapelig Kapittel/Artikkel/Konferanseartikkel
Vis sammendrag
The purpose of text sanitization is to edit text documents to mask text spans that may directly or indirectly reveal personal information. An important problem in text sanitization is to find less specific, yet still informative replacements for each text span to mask. We present an approach to generate possible replacements using a combination of heuristic rules and an ontology derived
from Wikidata. Those replacement options are hierarchically structured and cover various types of personal identifiers. Using this approach, we extend a recently released text sanitization dataset with manually selected replacements. The outcome of this data collection shows that the approach is able to suggest appropriate replacement options for most text spans.
Korsvoll, Nils Hallvard; Lison, Pierre; Reinertsen, Hilde; Elken, Mari; Korne, Haley De; Hansen, Kai Arne og Danbolt, Bjørn Kristian. (2023).
Fire tiltak for en bedre språkpolitikk i akademia.
NVA
Kronikk
Lison, Pierre og Kennington, Casey. (2023).
Who's in Charge? Roles and Responsibilities of Decision-Making Components in Conversational Robots. CUI@HRI
NVA
Vitenskapelig foredrag
Høst, Anders Mølmen; Lison, Pierre og Moonen, Leon. (2023).
Constructing a Knowledge Graph from Textual Descriptions of Software Vulnerabilities in the National Vulnerability Database.
Vis sammendrag
Knowledge graphs have shown promise for several cybersecurity tasks, such as vulnerability assessment and threat analysis. In this work, we present a new method for constructing a vulnerability knowledge graph from information in the National Vulnerability Database (NVD). Our approach combines named entity recognition (NER), relation extraction (RE), and entity prediction using a combination of neural models, heuristic rules, and knowledge graph embeddings. We demonstrate how our method helps to fix missing entities in
knowledge graphs used for cybersecurity and evaluate the performance.
Walker, Nicholas Thomas og Lison, Pierre. (2023).
GraphWOZ: Dialogue Management with Conversational Knowledge Graphs. IWSDS 2023
NVA
Vitenskapelig foredrag
Barnes, Jeremy Claude; Touileb, Samia; Mæhlum, Petter og Lison, Pierre. (2023).
Identifying Token-Level Dialectal Features in Social Media.
Vis sammendrag
Dialectal variation is present in many human languages and is attracting a growing interest in NLP. Most previous work concentrated on either (1) classifying dialectal varieties at the document or sentence level or (2) performing standard NLP tasks on dialectal data. In this paper, we propose the novel task of token-level dialectal feature prediction. We present a set of fine-grained annotation guidelines for Norwegian dialects, expand a corpus of dialectal tweets, and manually annotate them using the introduced guidelines. Furthermore, to evaluate the learnability of our task, we conduct labeling experiments using a collection of baselines, weakly supervised and supervised sequence labeling models. The obtained results show that, despite the difficulty of the task and the scarcity of training data, many dialectal features can be predicted with reasonably high accuracy.
Walker, Nicholas Thomas; Ultes, Stefan og Lison, Pierre. (2023).
Retrieval-Augmented Neural Response Generation Using Logical Reasoning and Relevance Scoring.
Vis sammendrag
Constructing responses in task-oriented dialogue systems typically relies on information sources such the current dialogue state or external databases. This paper presents a novel approach to knowledge-grounded response generation
that combines retrieval-augmented language models with logical reasoning. The approach revolves around a knowledge graph representing the current dialogue state and background
information, and proceeds in three steps.
The knowledge graph is first enriched with logically
derived facts inferred using probabilistic logical programming. A neural model is then employed at each turn to score the conversational relevance of each node and edge
of this extended graph. Finally, the elements with highest relevance scores are converted to a natural language form, and are integrated into the prompt for the neural conversational model employed to generate the system response.
We investigate the benefits of the proposed approach
on two datasets (KVRET and Graph-WOZ) along with a human evaluation. Experimental results show that the combination of
(probabilistic) logical reasoning with conversational
relevance scoring does increase both the factuality and fluency of the responses.
Walker, Nicholas Thomas; Ultes, Stefan og Lison, Pierre. (2023).
A Graph-to-Text Approach to Knowledge-Grounded Response Generation in Human-Robot Interaction.
Vis sammendrag
Knowledge graphs are often used to represent structured information in a flexible and efficient manner, but their use in situated dialogue remains under-explored. This paper presents a novel conversational model for human--robot interaction that rests upon a graph-based representation of the dialogue state. The knowledge graph representing the dialogue state is continuously updated with new observations from the robot sensors, including linguistic, situated and multimodal inputs, and is further enriched by other modules, in particular for spatial understanding. The neural conversational model employed to respond to user utterances relies on a simple but effective graph-to-text mechanism that traverses the dialogue state graph and converts the traversals into a natural language form. This conversion of the state graph into text is performed using a set of parameterized functions, and the values for those parameters are optimized based on a small set of Wizard-of-Oz interactions. After this conversion, the text representation of the dialogue state graph is included as part of the prompt of a large language model used to decode the agent response. The proposed approach is empirically evaluated through a user study with a humanoid robot that acts as conversation partner to evaluate the impact of the graph-to-text mechanism on the response generation. After moving a robot along a tour of an indoor environment, participants interacted with the robot using spoken dialogue and evaluated how well the robot was able to answer questions about what the robot observed during the tour. User scores show a statistically significant improvement in the …
Engebretsen, Solveig; Løland, Anders og Lison, Pierre. (2023).
Alt du kan lære om statistisk modellering og maskinlæring på en dag. Norsk Regnesentral
NVA
Faglig foredrag
Engebretsen, Solveig; Løland, Anders og Lison, Pierre. (2023).
Alt du kan lære om statistisk modellering og maskinlæring på en dag. Norsk Regnesentral
NVA
Faglig foredrag
Dahl, Fredrik Andreas; Eikvil, Line; Tvete, Ingunn Fride; Lison, Pierre; Pilán, Ildikó; Fuglerud, Kristin Skeide og Leister, Wolfgang. (2023).
Helse-effektivisering - et mulig satsningsområde for NR.
NVA
Rapport
Walker, Nicholas Thomas; Dahl, Torbjørn og Lison, Pierre. (2022).
Dialogue Management as Graph Transformations.
Vis sammendrag
We present ongoing work on a new dialogue management framework using graphs as core representation for the current dialogue state. Dialogue management tasks such as state tracking and action selection are framed as sequences of graph transformations that repeatedly update this graph based on incoming observations. Those graph transformations are expressed using a graph query language, making it possible to specify all dialogue management operations through a unified, declarative syntax. We argue that graphs are particularly well suited to model the dialogue state of complex, open-ended domains. In contrast to traditional dialogue state representations that are limited to fixed, predefined slots, graphs can naturally express dialogue domains with rich relational structures and variable numbers of entities to track. We describe how dialogue state tracking and action selection can be modelled in such graph-centric view of dialogue management, using either handcrafted rules or data-driven models. We also briefly discuss how to account for some aspects of dialogue management such as uncertainties, incremental inputs and contextual knowledge. Finally, we describe a proof-of-concept study of this dialogue management framework in a human–robot interaction scenario.
Løland, Anders; Fuglerud, Kristin Skeide og Lison, Pierre. (2022).
Hva er universell utforming?
NVA
Programdeltagelse
Lund, Bjarte Aarmo; Sandsør, Astrid Marie Jorde og Lison, Pierre. (2022).
Problemer på kontoret: Alltid jeg som må trakte kaffe.
NVA
Kronikk
Hassan, Syed Zohaib; Salehi, Pegah; Røed, Ragnhild Klingenberg; Halvorsen, Pål; Baugerud, Gunn Astrid; Johnson, Miriam S.; Lison, Pierre; Riegler, Michael; Lamb, Michael E.; Griwodz, Carsten og Sabet, Saeed. (2022).
Towards an AI-driven talking avatar in virtual reality for investigative interviews of children.
Vis sammendrag
Artificial intelligence (AI) and gaming systems have advanced to the stage where the current models and technologies can be used to address real-world problems. The development of such systems comes with different challenges, e.g., most of them related to system performance, complexity and user testing. Using a virtual reality (VR) environment, we have designed and developed a game-like system aiming to mimic an abused child that can help to assist police and child protection service (CPS) personnel in interview training of maltreated children. Current research in this area points to the poor quality of conducted interviews, and emphasises the need for better training methods. Information obtained in these interviews is the core piece of evidence in the prosecution process. We utilised advanced dialogue models, talking visual avatars, and VR to build a virtual child avatar that can interact with users. We discuss our proposed architecture and the performance of the developed child avatar prototype, and we present the results from the user study conducted with CPS personnel. The user study investigates the users' perceived quality of experience (QoE) and their learning effects. Our study confirms that such a gaming system can increase the knowledge and skills of the users. We also benchmark and discuss the system performance aspects of the child avatar. Our results show that the proposed prototype works well in practice and is well received by the interview experts.
Lison, Pierre; Rots, Aike Peter og Korne, Haley De. (2022).
Hvilket fremmedspråk bør man lære seg i Google-oversettelsenes tidsalder?
NVA
Kronikk
Pilán, Ildikó; Lison, Pierre; Øvrelid, Lilja; Papadopoulou, Anthi; Sánchez, David og Batet, Montserrat. (2022).
The text anonymization benchmark (TAB): A dedicated corpus and evaluation framework for text anonymization.
Vis sammendrag
We present a novel benchmark and associated evaluation metrics for assessing the performance of text anonymization methods. Text anonymization, defined as the task of editing a text document to prevent the disclosure of personal information, currently suffers from a shortage of privacy-oriented annotated text resources, making it difficult to properly evaluate the level of privacy protection offered by various anonymization methods. This paper presents TAB (Text Anonymization Benchmark), a new, open-source annotated corpus developed to address this shortage. The corpus comprises 1,268 English-language court cases from the European Court of Human Rights (ECHR) enriched with comprehensive annotations about the personal information appearing in each document, including their semantic category, identifier type, confidential attributes, and co-reference relations. Compared with previous work, the TAB corpus is designed to go beyond traditional de-identification (which is limited to the detection of predefined semantic categories), and explicitly marks which text spans ought to be masked in order to conceal the identity of the person to be protected.
Along with presenting the corpus and its annotation layers, we also propose a set of evaluation metrics that are specifically tailored toward measuring the performance of text anonymization, both in terms of privacy protection and utility preservation. We illustrate the use of the benchmark and the proposed metrics by assessing the empirical performance of several baseline text anonymization models. The full corpus along with its privacy-oriented annotation guidelines, evaluation scripts, and baseline models are available on: https://github.com/NorskRegnesentral/text-anonymization-benchmark.
Papadopoulou, Anthi; Yu, Yunhao; Lison, Pierre og Øvrelid, Lilja. (2022).
Neural Text Sanitization with Explicit Measures of Privacy Risk.
NVA
Vitenskapelig Kapittel/Artikkel/Konferanseartikkel
Vis sammendrag
We present a novel approach for text sanitization, which is the task of editing a document to mask all (direct and indirect) personal identifiers and thereby conceal the identity of the individuals(s) mentioned in the text. In contrast to previous work, the approach relies on explicit measures of privacy risk, making it possible to explicitly control the trade-off between privacy protection and data utility. The approach proceeds in three steps. A neural, privacy-enhanced entity recognizer is first employed to detect and classify potential personal identifiers. We then determine which entities, or combination of entities, are likely to pose a re-identification risk through a range of privacy risk assessment measures. We present three such measures of privacy risk, respectively based on (1) span probabilities derived from a BERT language model, (2) web search queries and (3) a classifier trained on labelled data. Finally, a linear optimization solver decides which entities to mask to minimize the semantic loss while simultaneously ensuring that the estimated privacy risk remains under a given threshold. We evaluate the approach both in the absence and presence of manually annotated data. Our results highlight the potential of the approach, as well as issues specific types of personal data can introduce to the process.
Lison, Pierre. (2022).
Anonymization of sensitive information. BI Norwegian Business School
NVA
Faglig foredrag
Lison, Pierre. (2022).
Dis, c'est quoi l'intelligence artificielle?
NVA
Populærvitenskapelig bok
Vis sammendrag
L’intelligence artificielle est aujourd’hui sur toutes les lèvres. Certains y voient une révolution technologique qui viendra nous libérer de nombreuses tâches pénibles, répétitives ou dangereuses et transformera notre rapport au monde et au travail. Pour d’autres, elle représente au contraire un danger pour l’équilibre déjà fragile de nos sociétés, voire une menace existentielle pour l’humanité. Mais de quoi s’agit-il au juste ? Cet ouvrage aborde de manière pédagogique les idées-clés qui sous-tendent l’intelligence artificielle. Comment une machine peut-elle apprendre, raisonner et résoudre par elle-même des problèmes complexes ? Quelles sont les applications pratiques de l’intelligence artificielle ? Et en quoi l’intelligence d’une machine se différencie-t-elle du fonctionnement de notre cerveau humain ? Au fil d’une conversation, ce livre permet de mieux comprendre cette discipline au confluent de l’informatique, des mathématiques et des sciences cognitives.
Papadopoulou, Anthi; Lison, Pierre; Øvrelid, Lilja og Pilán, Ildikó. (2022).
Bootstrapping Text Anonymization Models with Distant Supervision.
NVA
Vitenskapelig Kapittel/Artikkel/Konferanseartikkel
Vis sammendrag
We propose a novel method to bootstrap text anonymization models based on distant supervision. Instead of requiring
manually labeled training data, the approach relies on a knowledge graph expressing the background information assumed to be publicly available about various individuals. This knowledge graph is employed to automatically annotate text documents including personal data about a subset of those individuals. More precisely, the method determines which text spans ought to be masked in order to guarantee k-anonymity, assuming an adversary with access to both the text documents and the background information expressed in the knowledge graph. The resulting collection of labeled documents is then used as training data to fine-tune a pre-trained language model for text anonymization. We illustrate this approach using a knowledge graph extracted from Wikidata and short biographical texts from Wikipedia. Evaluation results with a RoBERTa-based model and a manually annotated collection of 553 summaries showcase the potential of the approach, but also unveil a number of issues that may arise if the knowledge graph is noisy or incomplete. The results also illustrate that, contrary to most sequence labeling problems, the text anonymization task may admit several alternative solutions.
Lison, Pierre; Korsvoll, Nils Hallvard og Lundberg, Aase-Kristine Aasen. (2022).
Kjernekraft -er det farlig, eller er det fremtiden?
NVA
Kronikk
Manzanares-Salor, Benet; Sánchez, David og Lison, Pierre. (2022).
Automatic Evaluation of Disclosure Risks of Text Anonymization Methods.
Vis sammendrag
The standard approach to evaluate text anonymization methods consists of comparing their outcomes with the anonymization performed by human experts. The degree of privacy protection attained is then measured with the IR-based recall metric, which expresses the proportion of re-identifying terms that were correctly detected by the anonymization method. However, the use of recall to estimate the degree of privacy protection suffers from several limitations. The first is that it assigns a uniform weight to each re-identifying term, thereby ignoring the fact that some missed re-identifying terms may have a larger influence on the disclosure risk than others. Furthermore, IR-based metrics assume the existence of a single gold standard annotation. This assumption does not hold for text anonymization, where several maskings (each one encompassing a different combination of terms) could be equally valid to prevent disclosure. Finally, those metrics rely on manually anonymized datasets, which are inherently subjective and may be prone to various errors, omissions and inconsistencies. To tackle these issues, we propose an automatic re-identification attack for (anonymized) texts that provides a realistic assessment of disclosure risks. Our method follows a similar premise as the well-known record linkage methods employed to evaluate anonymized structured data, and leverages state-of-the-art deep learning language models to exploit the background knowledge available to potential attackers. We also report empirical evaluations of several well-known methods and tools for text anonymization. Results show significant re-identification risks for all methods, including also manual anonymization efforts.
Weitzenboeck, Emily Mary; Lison, Pierre; Cyndecka, Malgorzata Agnieszka og Langford, Malcolm. (2022).
The GDPR and Unstructured Data: Is Anonymisation Possible?
Vis sammendrag
Much of the legal and technical literature on data anonymization has focused on structured data such as tables. However, unstructured data such as text documents or images are far more common, and the legal requirements that must be fulfilled to properly anonymize such data formats remain unclear and underaddressed by the literature.
In the absence of a definition of the term ‘anonymous data’ in the General Data Protection Regulation (GDPR), we examine its antithesis—personal data—and the identifiability test in Recital 26 GDPR to understand what conditions must be in place for the anonymization of unstructured data.
This article examines the two contrasting approaches for determining identifiability that are prevalent today: (i) the risk-based approach and (ii) the strict approach in the Article 29 Working Party’s Opinion on Anonymization Techniques (WP 216).
Through two case studies, we illustrate the challenges encountered when trying to anonymize unstructured datasets. We show that, while the risk-based approach offers a more nuanced test consistent with the purposes of the GDPR, the strict approach of WP 216 makes anonymization of unstructured data virtually impossible as long as the original data continues to exist.
The concluding section considers the policy implications of the strict approach and technological developments that assist identification, and proposes a way forward.
Korsvoll, Nils Hallvard; Elken, Mari; Sandtorv, Alexander Harald og Lison, Pierre. (2021).
Welcome to Norway!
NVA
Kronikk
Lison, Pierre. (2021).
Språkteknologi: siste trender og vanlige fallgruver. Nasjonalbiblioteket
NVA
Faglig foredrag
Lison, Pierre; Bølstad, Jørgen og Kvellestad, Anders. (2021).
Forvirrende pandemistatistikk: Hva skal vi med logaritmer i grafer?
NVA
Kronikk
Jokinen, Kristiina; Heckmann, Martin; Lala, Dinesh og Lison, Pierre. (2021).
Proceedings of the 1st RobotDial Workshop on Dialogue Models for Human-Robot Interaction.
NVA
Rapport
Lison, Pierre. (2021).
Skweak: Weak Supervision Made Easy for NLP. Research Group Data Mining and Machine Learning, University of Vienna
NVA
Vitenskapelig foredrag
Lison, Pierre; Barnes, Jeremy og Hubin, Aliaksandr. (2021).
skweak: Weak Supervision Made Easy for NLP.
Vis sammendrag
We present skweak, a versatile, Python-based software toolkit enabling NLP developers to apply weak supervision to a wide range of NLP tasks. Weak supervision is an emerging machine learning paradigm based on a simple idea: instead of labelling data points by hand, we use labelling functions derived from domain knowledge to automatically obtain annotations for a given dataset. The resulting labels are then aggregated with a generative model that estimates the accuracy (and possible confusions) of each labelling function. The skweak toolkit makes it easy to implement a large spectrum of labelling functions (such as heuristics, gazetteers, neural models or linguistic constraints) on text data, apply them on a corpus, and aggregate their results in a fully unsupervised fashion. skweak is especially designed to facilitate the use of weak supervision for NLP tasks such as text classification and sequence labelling. We illustrate the use of skweak for NER and sentiment analysis. skweak is released under an open-source license and is available at https://github.com/NorskRegnesentral/skweak
Olsen, Joakim; Næss, Arild Brandrud og Lison, Pierre. (2021).
Assessing the Quality of Human-Generated Summaries with Weakly Supervised Learning.
NVA
Vitenskapelig Kapittel/Artikkel/Konferanseartikkel
Vis sammendrag
This paper explores how to automatically measure the quality of human-generated summaries, based on a Norwegian corpus
of real estate condition reports and their corresponding summaries. The proposed approach proceeds in two steps. First, the real estate reports and their associated summaries are automatically labelled using a set of heuristic rules gathered from human experts and aggregated using weak supervision. The aggregated labels are then employed to learn a neural model that takes a document and its summary as inputs and outputs a score reflecting the predicted
quality of the summary. The neural model maps the document and its summary to a shared “summary content space” and computes the cosine similarity between the two document embeddings to predict the final summary quality score. The best performance is achieved by a CNN-based model
with an accuracy (measured against the aggregated labels obtained via weak supervision) of 89.5%, compared to 72.6% for the best unsupervised model. Manual inspection of examples indicate that the weak supervision labels do capture important indicators of summary quality, but the correlation of those labels with human judgements remains to be validated. Our models of summary quality predict that approximately 30% of the real estate reports in the
corpus have a summary of poor quality.
Lison, Pierre; Pilán, Ildikó; Øvrelid, Lilja; Ruenes, David Sánchez og Batet, Montserrat. (2021).
Anonymisation Models for Text Data: State of the art, Challenges and Future Directions. Association for Computational Linguistics
NVA
Vitenskapelig foredrag
Vis sammendrag
This position paper investigates the problem of automated text anonymisation, which is a prerequisite for secure sharing of documents containing sensitive information about individuals. We summarise the key concepts behind text anonymisation and provide a review of current approaches. Anonymisation methods have so far been developed in two fields with little mutual interaction, namely natural language processing and privacy-preserving data publishing. Based on a case study, we outline the benefits and limitations of these approaches and discuss a number of open challenges, such as (1) how to account for multiple types of semantic inferences, (2) how to strike a balance between disclosure risk and data utility and (3) how to evaluate the quality of the resulting anonymisation. We lay out a case for moving beyond sequence labelling models and incorporate explicit measures of disclosure risk into the text anonymisation process.
Lison, Pierre; Pilán, Ildikó; Ruenes, David Sánchez; Batet, Montserrat og Øvrelid, Lilja. (2021).
Anonymisation Models for Text Data: State of the art, Challenges and Future Directions. NAACL 2021 workshop
NVA
Vitenskapelig foredrag
Lison, Pierre; Barnes, Jeremy Claude og Hubin, Aliaksandr. (2021).
skweak: weak supervision made easy for NLP. Association for Computational Linguistics
NVA
poster
Vis sammendrag
We present skweak, a versatile, Python-based software toolkit enabling NLP developers to apply weak supervision to a wide range of NLP tasks. Weak supervision is an emerging machine learning paradigm based on a simple idea: instead of labelling data points by hand, we use labelling functions derived from domain knowledge to automatically obtain annotations for a given dataset. The resulting labels are then aggregated with a generative model that estimates the accuracy (and possible confusions) of each labelling function. The skweak toolkit makes it easy to implement a large spectrum of labelling functions (such as heuristics, gazetteers, neural models or linguistic constraints) on text data, apply them on a corpus, and aggregate their results in a fully unsupervised fashion. skweak is especially designed to facilitate the use of weak supervision for NLP tasks such as text classification and sequence labelling. We illustrate the use of skweak for NER and sentiment analysis. skweak is released under an open-source license and is available at https://github.com/NorskRegnesentral/skweak
Lison, Pierre; Pilán, Ildikó; Sánchez, David; Batet, Montserrat og Øvrelid, Lilja. (2021).
Anonymisation Models for Text Data: State of the art, Challenges and Future Directions.
Vis sammendrag
This position paper investigates the problem of automated text anonymisation, which is a prerequisite for secure sharing of documents containing sensitive information about individuals. We summarise the key concepts behind text anonymisation and provide a review of current approaches. Anonymisation methods have so far been developed in two fields with little mutual interaction, namely natural language processing and privacy-preserving data publishing. Based on a case study, we outline the benefits and limitations of these approaches and discuss a number of open challenges, such as (1) how to account for multiple types of semantic inferences, (2) how to strike a balance between disclosure risk and data utility and (3) how to evaluate the quality of the resulting anonymisation. We lay out a case for moving beyond sequence labelling models and incorporate explicit measures of disclosure risk into the text anonymisation process.
Redelmeier, Annabelle Alice; Lison, Pierre; Løland, Anders og Pilán, Ildikó. (2021).
Predicting insurance fraud with the help of a sentiment analysis model.
NVA
Rapport
Løland, Anders; Lison, Pierre og Falkum, Ingrid Lossius. (2020).
Episode 6: Kan språkteknologi virkelig forstå språk? Med Ingrid Lossius Falkum og Pierre Lison.
NVA
Programdeltagelse
Løland, Anders og Lison, Pierre. (2020).
Episode 5: Hva er språkteknologi (eller NLP)? Med Pierre Lison.
NVA
Programdeltagelse
Lison, Pierre. (2020).
Developing NLP models without labelled data using weak supervision. Norsk Forening for Kvantitativ Finans
NVA
Faglig foredrag
Lison, Pierre; Barnes, Jeremy; Hubin, Aliaksandr og Touileb, Samia. (2020).
Named Entity Recognition without Labelled Data: A Weak Supervision Approach.
Vis sammendrag
Named Entity Recognition (NER) performance often degrades rapidly when applied to target domains that differ from the texts observed during training. When in-domain labelled data is available, transfer learning techniques can be used to adapt existing NER models to the target domain. But what should one do when there is no hand-labelled data for the target domain? This paper presents a simple but powerful approach to learn NER models in the absence of labelled data through weak supervision. The approach relies on a broad spectrum of labelling functions to automatically annotate texts from the target domain. These annotations are then merged together using a hidden Markov model which captures the varying accuracies and confusions of the labelling functions. A sequence labelling model can finally be trained on the basis of this unified annotation. We evaluate the approach on two English datasets (CoNLL 2003 and news articles from Reuters and Bloomberg) and demonstrate an improvement of about 7 percentage points in entity-level F1 scores compared to an out-of-domain neural NER model.
Riegler, Michael; Lison, Pierre; Strümke, Inga og Løland, Anders. (2020).
For enkelt om kunstig intelligens: – Diskriminerende og fordomsfull AI er ikke alltid lett å løse.
NVA
Leserinnlegg
Lison, Pierre og Falkum, Ingrid Lossius. (2020).
Kan kunstig intelligens "forstå" språk?
NVA
Populærvitenskapelig artikkel
Lison, Pierre. (2020).
Named Entity Recognition without Labelled Data: A Weak Supervision Approach. Association for Computational Linguistics
NVA
Vitenskapelig foredrag
Jang, Youngsoo; Lee, Jongmin; Park, Jaeyoung; Lee, Kyeng-Hun; Lison, Pierre og Kee-Eung, Kim. (2019).
PyOpenDial: A Python-based Domain-Independent Toolkit for Developing Spoken Dialogue Systems with Probabilistic Rules.
Lison, Pierre. (2019).
Data-driven models of reputation for cybersecurity. Tekna Big Data
NVA
Vitenskapelig foredrag
Lison, Pierre. (2019).
Dialogue Modelling: Small data, Big data. Young Researcher's Roundtable on Spoken Dialogue Systems
NVA
Vitenskapelig foredrag
Lison, Pierre. (2019).
Modélisation du dialogue: contrôle du dialogue et corpus multilingues. Laboratoire Parole et Language
NVA
Vitenskapelig foredrag
Lison, Pierre. (2019).
Modellering av omdømme i cybersikkerhet med nevralske nettverk. Norsk informasjonssikkerhetsforum
NVA
Faglig foredrag
Lison, Pierre. (2018).
Data-driven models of reputation in cyber-security.
NVA
Vitenskapelig foredrag
Vis sammendrag
In this talk, I will present our work on developing data-driven, predictive models of reputation (such as benign or malicious) for end-point hosts. I'll focus on two particular questions:
1) Malware often relies on so-called domain-generation algorithms (DGAs) to produce "fake" domain names that are used to connect compromised hosts with a command-and-control server. Many types of DGAs are been developed, from simple hashing techniques to more sophisticated approaches based on wordlists. I will show that these malware-generated domain names can be detected through recurrent neural networks such as LSTMs or GRUs.
2) The second part of the talk will focus on neural models of traffic reputation learned from passive DNS data. Passive DNS data are collections of inter-server DNS queries captured by sensors distributed on the network. This data is a goldmine for predicting whether a given domain name or IP address is likely to be benign or malicious. I will describe a deep neural architecture that predicts the reputation of end-point hosts with high accuracy. The neural model is trained on a large passive DNS dataset (745 million entries) and relies on a broad range of features extracted from the DNs graph.
Lison, Pierre; Tiedemann, Jörg og Kouylekov, Milen. (2018).
OpenSubtitles 2018: Statistical Rescoring of Sentence Alignments in Large, Noisy Parallel Corpora. Norsk Regnesentral
NVA
Vitenskapelig foredrag
Vis sammendrag
Movie and TV subtitles are a highly valuable resource for the compilation of parallel corpora thanks to their availability in large numbers and across many languages. However, the quality of the resulting sentence alignments is often lower than for other parallel corpora. This paper presents a new major release of the OpenSubtitles collection of parallel corpora, which is extracted from a total of 3.7 million subtitles spread over 60 languages. In addition to a substantial increase in the corpus size (about 30 % compared to the previous version), this new release associates explicit quality scores to each sentence alignment. These scores are determined by a statistical regression model based on simple language-independent features and estimated on a small sample of aligned sentence pairs. Evaluation results show that the model is able predict lexical translation probabilities with a root mean square error of 0.07 (coefficient of determination R2 = 0.47). Based on the scores produced by this regression model, the parallel corpora can be filtered to prune out alignments with a score below a given threshold
Lison, Pierre. (2018).
SAFERS: Talegjenkjenning og maskinlæring for nødmeldetjenester. CSAM Health
NVA
Faglig foredrag
Lison, Pierre. (2018).
Modélisation du dialogue : systèmes de dialogue parlé et corpus multilingues. Center for Natural Language Processing, ULouvain
NVA
Vitenskapelig foredrag
Lison, Pierre og Dogruöz, Sega. (2018).
Detecting Machine-translated Subtitles in Large Parallel Corpora. BUCC scientific committee
Vis sammendrag
Parallel corpora extracted from online repositories of movie and TV subtitles are employed in a wide range of NLP applications, from language modelling to machine translation and dialogue systems. However, the subtitles uploaded in such repositories exhibit varying levels of quality. A particularly difficult problem stems from the fact that a substantial number of these subtitles are not written by human subtitlers but are simply generated through the use of online translation engines. This paper investigates whether these machine-generated subtitles can be detected automatically using a combination of linguistic and extra-linguistic features. We show that a feedforward neural network trained on a small dataset of subtitles can detect machine-generated subtitles with a F1-score of 0.64. Furthermore, applying this detection model on an unlabelled sample of subtitles allows us to provide a statistical estimate for the proportion of subtitles that are machine-translated (or are at least of very low quality) in the full corpus.
Lison, Pierre og Dogruöz, A. Seza. (2018).
Detecting Machine-translated Documents in Large Parallel Corpora.
NVA
Vitenskapelig Kapittel/Artikkel/Konferanseartikkel
Vis sammendrag
Parallel corpora extracted from online repositories of movie and TV subtitles are employed in a wide range of NLP applications, from language modelling to machine translation and dialogue systems. However, the subtitles uploaded in such repositories exhibit varying levels of quality. A particularly difficult problem stems from the fact that a substantial number of these subtitles are not written by human subtitlers but are simply generated through the use of online translation engines. This paper investigates whether these machine-generated subtitles can be detected automatically using a combination of linguistic and extra-linguistic features. We show that a feedforward neural network trained on a small dataset of subtitles can detect machine-generated subtitles with a F1-score of 0.64. Furthermore, applying this detection model on an unlabelled sample of subtitles allows us to provide a statistical estimate for the proportion of subtitles that are machine-translated (or are at least of very low quality) in the full corpus.
Lison, Pierre; Tiedemann, Jörg og Kouylekov, Milen. (2018).
OpenSubtitles 2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora.
NVA
Vitenskapelig Kapittel/Artikkel/Konferanseartikkel
Vis sammendrag
Movie and TV subtitles are a highly valuable resource for the compilation of parallel corpora thanks to their availability in large numbers and across many languages. However, the quality of the resulting sentence alignments is often lower than for other parallel corpora. This paper presents a new major release of the OpenSubtitles collection of parallel corpora, which is extracted from a total of 3.7 million subtitles spread over 60 languages. In addition to a substantial increase in the corpus size (about 30 % compared to the previous version), this new release associates explicit quality scores to each sentence alignment. These scores are determined by a statistical regression model based on simple language-independent features and estimated on a small sample of aligned sentence pairs. Evaluation results show that the model is able predict lexical translation probabilities with a root mean square error of 0.07 (coefficient of determination R2 = 0.47). Based on the scores produced by this regression model, the parallel corpora can be filtered to prune out alignments with a score below a given threshold.
Lison, Pierre. (2018).
Neural models for predicting the reputation of end-point hosts. IFI, UiO
NVA
Vitenskapelig foredrag
Lison, Pierre og Mavroeidis, Vasileios. (2017).
Automatic Detection of Malware-Generated Domains with Recurrent Neural Models.
NVA
Vitenskapelig artikkel
Vis sammendrag
Modern malware families often rely on domain-generation algorithms (DGAs) to determine rendezvous points to their command-and-control server. Traditional defence strategies (such as blacklisting domains or IP addresses) are inadequate against such techniques due to the large and continuously changing list of domains produced by these algorithms. This paper demonstrates that a machine learning approach based on recurrent neural networks is able to detect domain names generated by DGAs with high precision. The neural models are estimated on a large training set of domains generated by various malwares. Experimental results show that this data-driven approach can detect malware-generated domain names with a F1 score of 0.971. To put it differently, the model can automatically detect 93 % of malware-generated domain names for a false positive rate of 1:100.
Lison, Pierre og Kutuzov, Andrei. (2017).
Redefining Context Windows for Word Embedding Models: An Experimental Study.
Vis sammendrag
Distributional semantic models learn vector
representations of words through the
contexts they occur in. Although the
choice of context (which often takes the
form of a sliding window) has a direct influence
on the resulting embeddings, the
exact role of this model component is still
not fully understood. This paper presents
a systematic analysis of context windows
based on a set of four distinct hyperparameters.
We train continuous Skip-
Gram models on two English-language
corpora for various combinations of these
hyper-parameters, and evaluate them on
both lexical similarity and analogy tasks.
Notable experimental results are the positive
impact of cross-sentential contexts
and the surprisingly good performance of
right-context windows.
Lison, Pierre og Mavroeidis, Vasileios. (2017).
Neural Reputation Models learned from Passive DNS data.
Vis sammendrag
Blacklists and whitelists are often employed to filter outgoing and incoming traffic on computer networks. One central function of these lists is to mitigate the security risks posed by malware threats by associating a reputation (for instance benign or malicious) to end-point hosts. The creation and maintenance of these lists is a complex and time-consuming process for security experts. As a consequence, blacklists and whitelists are prone to various errors, inconsistencies and omissions, as only a tiny fraction of end-point hosts are effectively covered by the reputation lists. In this paper, we present a machine learning model that is able to automatically detect whether domain names and IP addresses are benign, malicious or sinkholes. The model relies on a deep neural architecture and is trained on a large passive DNS database. Evaluation results demonstrate the effectiveness of the approach, as the model is able to detect malicious DNS records with a F1 score of 0.96. In other words, the model is able to detect 95 % of the malicious hosts with a false positive rate of 1:1000.
Lison, Pierre. (2017).
Automatic Detection of Malware-Generated Domains with Recurrent Neural Models. NISK organisation committee
NVA
Vitenskapelig foredrag
Vis sammendrag
Modern malware families often rely on domain-generation algorithms
(DGAs) to determine rendezvous points to their command-and-control
server. Traditional defence strategies (such as blacklisting domains or IP
addresses) are inadequate against such techniques due to the large and
continuously changing list of domains produced by these algorithms. This
paper demonstrates that a machine learning approach based on recurrent
neural networks is able to detect domain names generated by DGAs with
high precision. The neural models are estimated on a large training set of
domains generated by various malwares. Experimental results show that this
data-driven approach can detect malware-generated domain names with a F1
score of 0.971. To put it differently, the model can automatically detect 93 %
of malware-generated domain names for a false positive rate of 1:100.
Lison, Pierre. (2017).
SAFERS - Speech Analytics for Emergency Response Services.
Kan taleteknologi og maskinlæring brukes for å effektivisere nødmeldetjenester? AmbulanseForum
NVA
Faglig foredrag
Lison, Pierre. (2017).
Neural Reputation Models learned from Passive DNS Data. IEEE Big Data 2017
NVA
Vitenskapelig foredrag
Lison, Pierre. (2017).
Opptreden i God Morgen Norge (TV2) for å vise Lenny roboten som ble brukt ved Forskningstorget.
NVA
Programdeltagelse
Lison, Pierre og Bibauw, Serge. (2017).
Not all dialogues are created equal: instance weighting for neural conversational models.
NVA
Vitenskapelig foredrag
Lison, Pierre og Kennington, Casey. (2017).
Incremental Processing for Neural Conversational Models.
Vis sammendrag
We present a simple approach to adapt neural conversation models to incremental processing. The approach is validated with a proof-of-concept experiment in a visual reference resolution task.
Lison, Pierre og Kennington, Casey. (2017).
Incremental Processing for Neural Conversational Models.
NVA
poster
Vis sammendrag
We presented a simple approach to make neural dialogue models 'incremental' - that is, able to operate on incremental units instead of on complete utterances. The model can handle insertions, commit and revoke operations as well as incremental units associated with probabilities. A proof-of-concept experiment on a visual reference resolution task shows the promise of the approach.
Lison, Pierre og Bibauw, Serge. (2017).
Not All Dialogues are Created Equal: Instance Weighting for Neural Conversational Models.
Vis sammendrag
Neural conversational models require substantial amounts of dialogue data for their parameter estimation and are therefore usually learned on large corpora such as chat forums or movie subtitles. These corpora are, however, often challenging to work with, notably due to their frequent lack of turn segmentation and the presence of multiple references external to the dialogue itself. This paper shows that these challenges can be mitigated by adding a weighting model into the architecture. The weighting model, which is itself estimated from dialogue data, associates each training example to a numerical weight that reflects its intrinsic quality for dialogue modelling. At training time, these sample weights are included into the empirical loss to be minimised. Evaluation results on retrieval-based models trained on movie and TV subtitles demonstrate that the inclusion of such a weighting model improves the model performance on unsupervised metrics.
Lison, Pierre. (2016).
Automatic Turn Segmentation for Movie and TV Subtitles.
NVA
Vitenskapelig foredrag
Lison, Pierre og Meena, Raveesh. (2016).
Automatic Turn Segmentation of Movie and TV Subtitles.
Vis sammendrag
Movie and TV subtitles contain large amounts of conversational material, but lack an explicit turn structure. This paper present a data-driven approach to the segmentation of subtitles into dialogue turns. Training data is first extracted by aligning subtitles with transcripts in order to obtain speaker labels. This data is then used to build a classifier whose task is to determine whether two consecutive sentences are part of the same dialogue turn. The approach relies on linguistic, visual and timing features extracted from the subtitles themselves and does not require access to the audiovisual material -- although speaker diarization can be exploited when audio data is available. The approach also exploits alignments with related subtitles in other languages to further improve the classification performance. The classifier achieves an accuracy of 78% on a held-out test set. A follow-up annotation experiment demonstrates that this task is also difficult for human annotators.