| Abstract [eng] |
Digital textual information is growing at a rapid pace. Effective automatic methods for understanding and processing such information are required, typically relying on the transformation of text into numerical representations. The emergence of Transformer-based models has created new challenges and opportunities for obtaining such representations. This dissertation investigates how Transformer models can be utilized for extracting text representations. For the English language, unsupervised representation extraction methods from pretrained Transformer models are explored for three groups of representation evaluation tasks: semantic textual similarity, short text clustering, and classification. For the Lithuanian language, the fine-tuning of Transformer models for grammatical error correction and abstractive summarization tasks is investigated, improving the original text itself. The dissertation presents methods that enable the extraction of improved representations from pretrained Transformer models. In addition, the use of a random vector model as a baseline method for future research is proposed. The first Transformer-based models for Lithuanian grammatical error correction and abstractive summarization are presented. The dissertation demonstrates that pretrained Transformer model weights can be effectively utilized to obtain high-quality representations without additional training, and how these models can be applied to lower-resource languages. |
