Code to get embeddings from wikipedia with BERT

To install dependencies run:

$ conda env create -f config/environment.yml

And then install the appropriate version of pytorch, transformers and download spacy dependencies:

$ conda install -y pytorch torchvision cudatoolkit=10.1 -c pytorch
$ # conda install pytorch torchvision cpuonly -c pytorch
$ pip install transformers

To get and tokenize data, use the github repository tpimentelms/wiki-tokenizer. Then put the parsed.txt files in path data/<language-name>/wiki.txt.

Use the Makefile to get a language's embeddings:

$ make LANGUAGE=id