All contextualized models show statistically significant improvement on Word2Vec (in all cases, p < 0.0005); and very similar results emerge regarding TransE, since the difference in scores is statistically significant for all models (BERT base: p = 0.0021, BERT large: p = 0.0004, LUKE base: p = 0.0054, LUKE large: p = 0.0085) but ELMO, which is approaching significance ( p = 0.057).
When comparing contextualized models with one another, statistical significance tests do not provide any p -value below 0.05, indicating that no reliable difference among the performances of these models exist; nevertheless, it should be noticed that BERT large is not only the model showing the best average results, but also the only model to get close to significance against other contextualized models (against BERT base: p = 0.064; against ELMO: p = 0.088; against LUKE large and base: p = 0.132), while the p-values for comparisons involving the other contextualized models are well above 0.3.