Exploring Linguistic Parameters in LMs

This work is an attempt to frame Principles and Parameters theory (Yang et al 2017) within the BabyLM Multilingual perspective. First, we compared tokenization and architectural strategies that make the incremental nature of language acquisition and processing more or less transparent (recurrent versus attention-based architectures). We then assessed our trained models with respect to specific linguistic generalizations, including morphological decomposition, learning trajectories, and binary parametric linguistic options. We observed that (i) morphologically informed tokenizers yield consistent benefits, both in terms of learning trajectories and performance on downstream minimal-pair tasks; (ii) recurrent architectures are substantially more data-efficient, converging earlier and achieving slightly better results on downstream evaluation tasks; and (iii) the parameter set-up objective leads to modest improvements and not in all the expected linguistic domains.