Try out different word embeddings for BERT intrinsic evaluation

[siemdejong]

Research question
The last hidden layer of BERT is best suited for contextualized text embeddings.

Hypothesis
It is the layer where the structure is best defined, considering all previous relations in the other 11 layers.

Method

1. Instantiate pretrained ClinialBERT
2. Gather a dataset of medical terms with different classes. E.g. all brain locations, but locations are grouped by occurrence of tumours in those regions.
3. Generate embeddings from layers as proposed in https://jalammar.github.io/illustrated-bert/
4. Intrinsic evaluation per embedding strategy. Evaluation measure tbd

Why is this experiment worthwhile?
Papers report different accuracies when using different embedding strategies from pretrained models (ref!).