WANNA BE ON TOP? The Hyperparameter Search for Semantic Change's Next Top Model

Abstract

Lexical semantic change (LSC) detection through the use of diachronic corpora and computational methods continues to be a prevalent research area in language change (Tahmasebi et al., 2018). However, there has not yet been (to the best of our knowledge) extensive work further examining the models being trained and creating a foundation for what hyperparameter settings yield the best results. In this thesis, a large-scale hyperparameter search is conducted using the SemEval-2020 Task 1 dataset that includes English, German, Swedish, and Latin. Alongside model hyperparameters, different algorithms (Word2Vec and FastText) and alignment methods (Orthogonal Procrustes and Incremental Training) were also included. The hyperparameters evaluated are: number of training epochs, vector dimension, frequency threshold, and shared vocabulary size for the Orthogonal Procrustes alignment method. By amalgamating all of the results and assessing how model performance is affected if one hyperparameter is changed, considerations that must be made before training a model were substantiated. This research concludes that improvements in performance significantly decreases after 50 epochs during training and that the typical choice of 300 dimensions for vectors (based on English best practices in NLP) does not necessarily apply to other languages. It is also shown that choices in vector dimension, frequency threshold, and shared vocabulary size depend on the language in question, corpus size, and text genre composition.