Publications

2023

1. The state of human-centered NLP technology for fact-checking

   Anubrata Das, Houjiang Liu, Venelin Kovatchev, and 1 more author

   Information Processing & Management, 2023

   Abs

   Misinformation threatens modern society by promoting distrust in science, changing narratives in public health, heightening social polarization, and disrupting democratic elections and financial markets, among a myriad of other societal harms. To address this, a growing cadre of professional fact-checkers and journalists provide high-quality investigations into purported facts. However, these largely manual efforts have struggled to match the enormous scale of the problem. In response, a growing body of Natural Language Processing (NLP) technologies have been proposed for more scalable fact-checking. Despite tremendous growth in such research, however, practical adoption of NLP technologies for fact-checking still remains in its infancy today. In this work, we review the capabilities and limitations of the current NLP technologies for fact-checking. Our particular focus is to further chart the design space for how these technologies can be harnessed and refined in order to better meet the needs of human fact-checkers. To do so, we review key aspects of NLP-based fact-checking: task formulation, dataset construction, modeling, and human-centered strategies, such as explainable models and human-in-the-loop approaches. Next, we review the efficacy of applying NLP-based fact-checking tools to assist human fact-checkers. We recommend that future research include collaboration with fact-checker stakeholders early on in NLP research, as well as incorporation of human-centered design practices in model development, in order to further guide technology development for human use and practical adoption. Finally, we advocate for more research on benchmark development supporting extrinsic evaluation of human-centered fact-checking technologies.

2. Machine Learning and Deep Learning Systems for Automated Measurement of “Advanced” Theory of Mind: Reliability and Validity in Children and Adolescents

   Rory T. Devine, Venelin Kovatchev, Imogen Grumley Traynor, and 2 more authors

   Psychological Assessment, 2023

2022

1. longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks.

   Venelin Kovatchev, Trina Chatterjee, Venkata S Govindarajan, and 9 more authors

   In Proceedings of the First Workshop on Dynamic Adversarial Data Collection, Jul 2022

   Abs

   Developing methods to adversarially challenge NLP systems is a promising avenue for improving both model performance and interpretability. Here, we describe the approach of the team “longhorns” on Task 1 of the The First Workshop on Dynamic Adversarial Data Collection (DADC), which asked teams to manually fool a model on an Extractive Question Answering task. Our team finished first (pending validation), with a model error rate of 62%. We advocate for a systematic, linguistically informed approach to formulating adversarial questions, and we describe the results of our pilot experiments, as well as our official submission.

2. ProtoTEx: Explaining Model Decisions with Prototype Tensors

   Anubrata Das, Chitrank Gupta, Venelin Kovatchev, and 2 more authors

   In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Jul 2022
3. InferES : A Natural Language Inference Corpus for Spanish Featuring Negation-Based Contrastive and Adversarial Examples

   Venelin Kovatchev, and Mariona Taulé

   In Proceedings of the 29th International Conference on Computational Linguistics, Oct 2022

   Abs

   In this paper we present InferES - an original corpus for Natural Language Inference (NLI) in European Spanish. We propose, implement, and analyze a variety of corpus-creating strategies utilizing expert linguists and crowd workers. The objectives behind InferES are to provide high-quality data, and at the same time to facilitate the systematic evaluation of automated systems. Specifically, we focus on measuring and improving the performance of machine learning systems on negation-based adversarial examples and their ability to generalize across out-of-distribution topics. We train two transformer models on InferES (8,055 gold examples) in a variety of scenarios. Our best model obtains 72.8% accuracy, leaving a lot of room for improvement. The “hypothesis-only” baseline performs only 2%-5% higher than majority, indicating much fewer annotation artifacts than prior work. We show that models trained on InferES generalize very well across topics (both in- and out-of-distribution) and perform moderately well on negation-based adversarial examples.

2021

1. Can vectors read minds better than experts? Comparing data augmentation strategies for the automated scoring of children’s mindreading ability

   Venelin Kovatchev, Phillip Smith, Mark Lee, and 1 more author

   In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Oct 2021

2020

1. An Analysis of Natural Language Inference Benchmarks through the Lens of Negation

   Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, and 3 more authors

   In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Nov 2020
2. Decomposing and Comparing Meaning Relations: Paraphrasing, Textual Entailment, Contradiction, and Specificity

   Venelin Kovatchev, Darina Gold, M. Antonia Marti, and 2 more authors

   In Proceedings of the 12th Language Resources and Evaluation Conference, May 2020

   Abs

   In this paper, we present a methodology for decomposing and comparing multiple meaning relations (paraphrasing, textual entailment, contradiction, and specificity). The methodology includes SHARel - a new typology that consists of 26 linguistic and 8 reason-based categories. We use the typology to annotate a corpus of 520 sentence pairs in English and we demonstrate that unlike previous typologies, SHARel can be applied to all relations of interest with a high inter-annotator agreement. We analyze and compare the frequency and distribution of the linguistic and reason-based phenomena involved in paraphrasing, textual entailment, contradiction, and specificity. This comparison allows for a much more in-depth analysis of the workings of the individual relations and the way they interact and compare with each other. We release all resources (typology, annotation guidelines, and annotated corpus) to the community.

3. “What is on your mind?” Automated Scoring of Mindreading in Childhood and Early Adolescence

   Venelin Kovatchev, Phillip Smith, Mark Lee, and 3 more authors

   In Proceedings of the 28th International Conference on Computational Linguistics, May 2020

2019

1. Annotating and analyzing the interactions between meaning relations

   Darina Gold, Venelin Kovatchev, and Torsten Zesch

   In Proceedings of the 13th Linguistic Annotation Workshop, Aug 2019
2. A Qualitative Evaluation Framework for Paraphrase Identification

   Venelin Kovatchev, M. Antonia Marti, Maria Salamo, and 1 more author

   In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), Sep 2019

   Abs

   In this paper, we present a new approach for the evaluation, error analysis, and interpretation of supervised and unsupervised Paraphrase Identification (PI) systems. Our evaluation framework makes use of a PI corpus annotated with linguistic phenomena to provide a better understanding and interpretation of the performance of various PI systems. Our approach allows for a qualitative evaluation and comparison of the PI models using human interpretable categories. It does not require modification of the training objective of the systems and does not place additional burden on the developers. We replicate several popular supervised and unsupervised PI systems. Using our evaluation framework we show that: 1) Each system performs differently with respect to a set of linguistic phenomena and makes qualitatively different kinds of errors; 2) Some linguistic phenomena are more challenging than others across all systems.

2018

1. WARP-Text: a Web-Based Tool for Annotating Relationships between Pairs of Texts

   Venelin Kovatchev, M. Antònia Martı́, and Maria Salamó

   In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, Aug 2018

   Abs

   We present WARP-Text, an open-source web-based tool for annotating relationships between pairs of texts. WARP-Text supports multi-layer annotation and custom definitions of inter-textual and intra-textual relationships. Annotation can be performed at different granularity levels (such as sentences, phrases, or tokens). WARP-Text has an intuitive user-friendly interface both for project managers and annotators. WARP-Text fills a gap in the currently available NLP toolbox, as open-source alternatives for annotation of pairs of text are not readily available. WARP-Text has already been used in several annotation tasks and can be of interest to the researchers working in the areas of Paraphrasing, Entailment, Simplification, and Summarization, among others.

2. ETPC - A Paraphrase Identification Corpus Annotated with Extended Paraphrase Typology and Negation

   Venelin Kovatchev, M. Antònia Martı́, and Maria Salamó

   In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), May 2018