Automatic Detection of Lexical Loanwords in a Text Corpus

Abstract

In the context of globalization and the dynamic interaction between language speakers and representatives of diverse cultures, the incorporation of foreign lexical items into vocabulary systems has become a fundamental driver of linguistic development and enrichment. However, the exponential growth in textual data has rendered manual analysis and lexical unit identification increasingly inefficient and time-consuming. This necessitates the implementation of automated natural language processing (NLP) methods for loanword extraction. This paper aims to examine various approaches to automatic lexical item extraction from 1986 to the present, while also developing an algorithm to address this challenge. The material includes a collected corpus of 22348 English sentences parsed from the websites of 11 leading universities in Austria, Germany and Russia. To verify the results, 47 new sentences were used. Additionally, 1318 new sentences including German loanwords were generated using chatbots. As for the methods, the “bert-base-multilingual-cased model” was used in the study. Corpus was annotated with two tags indicating the presence/absence of a German loanword in a sentence. The model was then retrained on the corpus and on additionally generated sentences. The findings demonstrate that while contemporary methods achieve high accuracy rates, significant challenges persist in model performance across different language pairs and in overall efficiency enhancement. Furthermore, the study describes an algorithm for automatic extraction of German loanwords from English sentences utilizing the BERT large language model trained on a corpus of 900 texts. The model demonstrated robust performance, successfully identifying 30 out of 43 words of German origin.