TerseBERT

This repository contains information and code for TerseBERT, a pretrained language model created by fine-tuning BERT. TerseBERT is not only able to predict which word is most likely in a given context (like a regular language model), but if any word is necessary at all. It was created as a component of a text simplification solution described in the article Multi-Word Lexical Simplification presented at the COLING 2020 conference in Barcelona.

For example, consider the sentence The fat cat sat on the mat. If we mask the word mat and ask for the most likely predictions, both BERT and TerseBERT suggest floor, bed, table, etc. If we mask the word fat, BERT proposes black, white, big, while TerseBERT offers the same predictions, but also reports a high probability (80%) of [NONE] token. This indicates the sentence is likely to have no words in the selected location, as we can simply say The cat sat on the mat.

This document is a guide for obtaining, training and using a TerseBERT model. If you need any more information consult the paper or contact its authors!

The TerseBERT model trained for the study mentioned above is available in Hugging Face Transformers format for download here (1.3 GB). Provided you have PyTorch and NumPy installed, you can invoke TerseBERT in the following way:

import torch
import numpy as np
from transformers import BertTokenizer, BertModel, BertForMaskedLM

premodel='bert-large-uncased-whole-word-masking'
tokenizer = BertTokenizer.from_pretrained(premodel)
model_dict = torch.load("/PATH/TO/tersebert_pytorch_1_0.bin")
model = BertForMaskedLM.from_pretrained(pretrained_model_name_or_path=premodel, state_dict=model_dict)

sentence="The fat cat sat on the mat."
tokenized_text = tokenizer.tokenize(sentence)
masked_text=['[CLS]']+tokenized_text+['[SEP]']
masked_token=tokenized_text.index("fat")+1
masked_text[masked_token]='[MASK]'
indexed_tokens=tokenizer.convert_tokens_to_ids(masked_text)
segments_ids=[0]*len(masked_text)
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])
predictions = model(tokens_tensor, segments_tensors)
scores=predictions[0][0][masked_token].detach().numpy()
scores=np.exp(scores)/sum(np.exp(scores))
tops=(-scores).argsort()[0:10]
predicted_token = ['[NONE]' if i==1 else tokenizer.convert_ids_to_tokens([i])[0] for i in tops]
print(list(zip(predicted_token,scores[tops])))

You should expect the following output:

[('[NONE]', 0.80043215), ('black', 0.057732496), ('white', 0.036740497), ('big', 0.011116397), ('little', 0.005187636), ('gray', 0.0025780434), ('fat', 0.0025481516), ('house', 0.0021788846), ('old', 0.0021117083), ('giant', 0.001906771)]

To train your own TerseBERT, follow these steps:

1. Prepare a large corpus of documents for finetuning. We chose Wikipedia and used WikiExtractor to extract plain text, but you can use any source, e.g. with domain-specific documents.
2. Choose a BERT model to fine-tune (we used BERT-Large, Uncased (Whole Word Masking)) and modify its dictionary to include the [NONE] token: see vocab_none.txt.
3. Create pretraining data by using create_pretraining_data_none.py, a variant of BERT's create_pretraining_data.py modified to insert [NONE] in random places in a defined number (we used 5%).
4. Run pretraining in the usual way (in our case 5000 steps was enough).
5. Convert the returned model into Hugging Face format.

• Like the original BERT code, the modified pretraining script is licensed under Apache Licence 2.0.
• The pretrained model is released under the CC BY-NC-SA 4.0 licence.

Przybyła, P. and Shardlow, M., 2020. Multi-Word Lexical Simplification. In Proceedings of the 28th International Conference on Computational Linguistics (COLING 2020).

@inproceedings{plainifier,
    title = "Multi-Word Lexical Simplification",
    author = {Przyby{\l}a, Piotr and Shardlow, Matthew}",
    booktitle = {Proceedings of the 28th International Conference on Computational Linguistics (COLING 2020)},
    month = dec,
    year = "2020",
    address = "Barcelona, Spain",
    publisher = {International Committee on Computational Linguistics},
    pages = {1435--1446},
    url = {https://www.aclweb.org/anthology/2020.coling-main.0}
}