Testing Encoders

I wanted to get a feel for myself of how well Masked Language Models perform at uncovering masks. This is a fairly trivial exercise, but I wanted to do it myself to get a better intuition of relative performance. Maybe it’ll be helpful to you as well, code on the bottom.

The Tasks: Answer these questions:

Question	Expected Answer
The cat chased the [MASK].	mouse
The capital of [MASK] is Paris.	France
Bread is typically baked in an [MASK].	oven
When it’s raining, you use an [MASK] to stay dry.	umbrella
You brush your [MASK] to keep them clean.	teeth
Money doesn’t grow on [MASK].	trees
A doctor often works in a [MASK].	hospital
You don’t put metal in a [MASK].	microwave
When you’re sleepy, you go to [MASK].	bed
The [MASK] is the center of our solar system.	sun
People breathe [MASK] to live.	oxygen
Plants need [MASK] to grow.	water
The [MASK] is a natural satellite of the Earth.	moon
Cars usually run on [MASK].	gasoline
A fridge is used to keep food [MASK].	cold
To see stars, you look at the [MASK].	sky
Humans have [MASK] fingers on each hand.	five
Fish live in [MASK].	water
Fire needs [MASK] to burn.	oxygen
Birds use [MASK] to fly.	wings
The [MASK] rises in the East.	sun

Models tested include BERT, Roberta, Electra, DeBERTa, XLM_Roberta_Base, BERT_Large_Cased, Legal_BERT, InfoXLM_Large, and Albert_Base_V1. The results are displayed in tables for each question, comparing the top 5 predicted tokens and their scores from each model.

CODE:

https://colab.research.google.com/drive/1FRyFGc9R70xd7nLeu9pmNBrWc2G5wcuh?usp=sharing

Evaluation:

I’m underwhelmed by the models’ performance. I’m sure I could’ve tweaked things further to get better outcomes, but I still expected more. Roberta did reasonably well, but still not too exciting.