This repository contains data and code for SLOG, a Structural LOng-distance dependencies Generalization benchmark. SLOG is a semantic parsing dataset that extends the COGS benchmark (Kim and Linzen,2020) with 17 structural generalization tasks.
-
The training sets and generalization sets can be found under
/data, please note that access to the generalization sets is restricted with a password-protected Zip file, with the password being: SLOG. This precaution is to prevent the sets from being unintentionally included in the training data of LLMs, ensuring they remain valid tools for unbiased model evaluation. -
The third column of the generalization set files specifies the 17 generalization types. For a comprehensive description with illustrative examples, please refer to Table 2 in the paper.
-
The code for generating SLOG is under
/generation_scriptsdirectory.
On SLOG, we trained a vanilla Transformer from scratch /experiments/vanilla_transformer, finetuned a pretrained T5-base: /experiments/T5 and a pretrained LLaMa /experiments/llama. For hyperparameters and random seed details of each model, please see the respective directories.