clusters-in-language-evaluation

Code accompanying the ICLR 2023 publication On the Usefulness of Embeddings, Clusters and Strings for Text Generator Evaluation.

Create a conda environment with

$ conda env create -f environment.yml

Then activate the environment and install your appropriate version of PyTorch.

$ conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
$ pip install transformers

Then install the local version of mauve:

$ git submodule update --init --recursive
$ pip install -e ./mauve

Then build the library in the kenlm submodule

$ cd kenlm
$ mkdir -p build
$ cd build
$ cmake ..
$ make -j 4

And install it to be usable in python:

$ pip install https://github.com/kpu/kenlm/archive/master.zip

Finaly, install r-base, which is necessary for the probing analysis experiments:

$ conda install -c conda-forge r-base

To download the text data that will be needed for the experiments run:

$ make get_text MODEL=human
$ make get_text MODEL=<model>
$ make get_representations MODEL=human REPRESENTATIONS=<representations>
$ make get_representations MODEL=<model> REPRESENTATIONS=<representations>

where <model> is the model you want to analyse text from, and <representations> is the model from which you want the embeddings for evaluation: ['gpt2-small', 'gpt2', 'gpt2-large', 'gpt2-xl'].

To estimate the language models p_w and q_w on the data:

$ make train_text MODEL=human LANGMODEL=<langmodel>
$ make train_text MODEL=<model> LANGMODEL=<langmodel>
$ make eval_text MODEL=<model> LANGMODEL=<langmodel>

where <langmodel> can be one of an: ['ngram', 'lstm'].

Then, to compute mauve and other divergences on text distributions:

$ make analyse_texts MODEL=<model> LANGMODEL=<langmodel>

To fit k-means on the embeddings, and get clusters (for both the model and human text) run:

$ make get_clusters MODEL=<model> REPRESENTATIONS=<representations>

Then, to estimate the cluster distributions p_c and q_c and evaluate them on the data:

$ make train_clusters MODEL=<model> REPRESENTATIONS=<representations>
$ make eval_clusters MODEL=<model> REPRESENTATIONS=<representations>

To compute mauve and other divergences on cluster distributions:

$ make analyse_clusters MODEL=<model> REPRESENTATIONS=<representations>

Finally, after you are done evaluating text and clusters for all models of interest (small-117M, small-117M-p0.9, medium-345M, medium-345M-p0.9, large-762M, large-762M-p0.95, xl-1542M, xl-1542M-p0.95), compile all results by running:

$ make get_results

Now you can compare to human text by running:

$ make compare
$ make plot

The plots for this portion of the paper were made using R. In order to recreate them, you'll need to install R and the rpy2 python library

# install R using your preferred OS package manager
apt-get install r-base
# install python--R interface
pip install rpy2

pip install rpy2
python src/notebooks/get_news_articles.py

All probing experiments can then be performed by running codeblocks in src/notebooks/main.ipynb linearly

Here is the bibtex citation for our paper:

@inproceedings{pimentel2023on,
	title={On the Usefulness of Embeddings, Clusters and Strings for Text Generation Evaluation},
	author={Tiago Pimentel and Clara Isabel Meister and Ryan Cotterell},
	booktitle={The Eleventh International Conference on Learning Representations },
	year={2023},
	url={https://openreview.net/forum?id=bvpkw7UIRdU}
}

To ask questions or report problems, please open an issue.