The field of natural language processing (NLP) has adopted deep learning methods in the past 15 years. Nowadays the state-of-the-art in most NLP tasks is some kind of neural model, often the fine-tuned version of a pre-trained language model. The efficacy of these models is demonstrated on various English benchmarks and increasingly, other monolingual and multimultilingual benchmarks. In this thesis I explore the application of deep learning models on low level tasks, particularly morphosyntactic tasks in multiple languages.

This talk covers the main contributions of my thesis. The first part of this thesis explores the application of deep learning models for classical morphosyntactic tasks such as morpholog- ical inflection and generation in dozens of languages with special focus on Hungarian.

The section part of this thesis deals with pre-trained language models, mostly large language models from the BERT (Devlin et al., 2018) family. These models show excellent performance on various tasks in English and some high density languages. However, their evaluation in medium and low density languages is lacking. I present a methodology for generating morphosyntactic benchmarks in arbitrary languages and I analyze multiple BERT-like models in detail. My main tool for analysis is the probing methodology (Belinkov, 2021).

Video

Slides