VSE knjižnice (vzajemna bibliografsko-kataložna baza podatkov COBIB.SI)
  • Geographic adaptation of pretrained language models [Elektronski vir]
    Hofmann, Valentin ...
    While pretrained language models (PLMs) have been shown to possess a plethora of linguistic knowledge, the existing body of research has largely neglected extralinguistic knowledge, which is ... generally difficult to obtain by pretraining on text alone. Here, we contribute to closing this gap by examining geolinguistic knowledge, i.e., knowledge about geographic variation in language. We introduce geoadaptation, an intermediate training step that couples language modeling with geolocation prediction in a multi-task learning setup. We geoadapt four PLMs, covering language groups from three geographic areas, and evaluate them on five different tasks: fine-tuned (i.e., supervised) geolocation prediction, zero-shot (i.e., unsupervised) geolocation prediction, fine-tuned language identification, zero-shot language identification, and zero-shot prediction of dialect features. Geoadaptation is very successful at injecting geolinguistic knowledge into the PLMs: The geoadapted PLMs consistently outperform PLMs adapted using only language modeling (by especially wide margins on zero-shot prediction tasks), and we obtain new state-of-the-art results on two benchmarks for geolocation prediction and language identification. Furthermore, we show that the effectiveness of geoadaptation stems from its ability to geographically retrofit the representation space of the PLMs.
    Vir: Transactions of the Association for Computational Linguistics [Elektronski vir]. - ISSN 2307-387X (Vol. 12, 2024, str. 411-431)
    Vrsta gradiva - e-članek ; neleposlovje za odrasle
    Leto - 2024
    Jezik - angleški
    COBISS.SI-ID - 196896515

    Povezava(-e):

    https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00652/120648/Geographic-Adaptation-of-Pretrained-Language
    DOI

Vnos na polico