Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks

This repository is an official implementation of the ACL 2026 (Main) paper "Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks".

Note

Throughout this repository, we use the following placeholders to refer to specific paths:

$SCRATCH: Refers to the scratch directory allocated for your user on the computing cluster.
/path/to/l2t/repository: Refers to the root directory of this repository.
your_partition_name: Refers to the specific partition name of the computing cluster you are using. Please replace this placeholder with the actual partition name when submitting jobs.

We assume access to high-performance computing clusters with either AMD or NVIDIA GPUs with Singularity/Apptainer installed and SLURM as the job scheduler. Please adjust the instructions accordingly if you are using a different setup.

Installation

For AMD GPUs

We create two separate environments: one for pre-training and another for evaluation.

For pre-training

# Download the container (if not already done)
mkdir -p $SCRATCH/containers/
APPTAINER_CACHEDIR=$SCRATCH/containers/
export APPTAINER_CACHEDIR
apptainer pull --dir $SCRATCH/containers/ docker://rocm/pytorch:rocm6.3_ubuntu22.04_py3.10_pytorch_release_2.3.0
mkdir $DATA/containers

apptainer exec --fakeroot \
    --bind $SCRATCH:$SCRATCH \
    --rocm $SCRATCH/containers/pytorch_rocm6.3_ubuntu22.04_py3.10_pytorch_release_2.3.0.sif \
    /bin/bash

# Sanity checks
rocminfo
rocm-smi

# Set configurations
mkdir -p $SCRATCH/cache/
export TRANSFORMERS_VERBOSITY=debug
export HF_HOME=$SCRATCH/cache/
export HF_HUB_CACHE=$SCRATCH/cache/
export HF_DATASETS_CACHE=$SCRATCH/cache/
export HF_DATASETS_TRUST_REMOTE_CODE=true

# Create an env
mkdir -p $SCRATCH/envs/
python3 -m venv --system-site-packages $SCRATCH/envs/arr_2026_jan
source $SCRATCH/envs/arr_2026_jan/bin/activate

# Install packages
pip install transformers==4.49.0 datasets==3.6.0 peft==0.15.1 accelerate==1.6.0 scikit-learn==1.6.1 sentencepiece==0.2.1 tqdm protobuf tiktoken==0.12.0 nltk==3.9.2 zstandard==0.25.0 huggingface-hub blingfire
cd ~/src
git clone --depth 1 https://github.com/ROCm/flash-attention.git
cd flash-attention
MAX_JOBS=$((`nproc` - 1)) pip install -v . # This takes a super long time ~2 hours

deactivate
exit
exit

For evaluation

# Download the container (if not already done)
mkdir -p $SCRATCH/containers/
APPTAINER_CACHEDIR=$SCRATCH/containers/
export APPTAINER_CACHEDIR
apptainer pull --dir $SCRATCH/containers/ docker://rocm/pytorch:rocm6.3_ubuntu22.04_py3.10_pytorch_release_2.3.0
mkdir $DATA/containers

apptainer exec --fakeroot \
    --bind $SCRATCH:$SCRATCH \
    --rocm $SCRATCH/containers/pytorch_rocm6.3_ubuntu22.04_py3.10_pytorch_release_2.3.0.sif \
    /bin/bash

# Sanity checks
rocminfo
rocm-smi

# Set configurations
mkdir -p $SCRATCH/cache/
export TRANSFORMERS_VERBOSITY=debug
export HF_HOME=$SCRATCH/cache/
export HF_HUB_CACHE=$SCRATCH/cache/
export HF_DATASETS_CACHE=$SCRATCH/cache/
export HF_DATASETS_TRUST_REMOTE_CODE=true

# Create an env
mkdir -p $SCRATCH/envs/
python3 -m venv --system-site-packages $SCRATCH/envs/arr_2026_jan
source $SCRATCH/envs/arr_2026_jan_eval/bin/activate

# Install packages
pip install transformers==4.49.0 datasets==3.6.0 peft==0.15.1 accelerate==1.6.0 scikit-learn==1.6.1 sentencepiece==0.2.1 tqdm protobuf tiktoken==0.12.0 nltk==3.9.2 zstandard==0.25.0 huggingface-hub blingfire
cd ~/src
git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
pip install  ".[math,ifeval,sentencepiece]"
huggingface-cli login

deactivate
exit
exit

For NVIDIA GPUs

For pre-training 1B models, we use the NVIDIA PyTorch container with CUDA support. We did not use NVIDIA GPUs for evaluation.

#!/bin/bash

# Download the container (if not already done)
mkdir -p $SCRATCH/containers/
APPTAINER_CACHEDIR=$SCRATCH/containers/
export APPTAINER_CACHEDIR
apptainer pull --dir $SCRATCH/containers/ docker://nvcr.io/nvidia/pytorch:25.04-py3

# Enable the CUDA environment
apptainer exec \
    --bind $SCRATCH:$SCRATCH \
    --nv $SCRATCH/containers/pytorch_25.04-py3.sif \
    /bin/bash

mkdir -p $SCRATCH/envs/
python3 -m venv --system-site-packages $SCRATCH/envs/arr_2026_jan
source $SCRATCH/envs/arr_2026_jan/bin/activate

# Install packages
unset PIP_CONSTRAINT
pip install transformers==4.49.0 datasets==3.6.0 peft==0.15.1 accelerate==1.6.0 scikit-learn==1.6.1 sentencepiece==0.2.1 tqdm protobuf tiktoken==0.12.0 nltk==3.9.2 zstandard==0.25.0 huggingface-hub blingfire

deactivate
exit
exit

Pre-processing

Please use the following scripts to pre-process the data for different scenarios:

Scenario	Script	Description
Disjoint	script / Slurm script	Pre-process data for the Disjoint scenario. Make sure to run this script with a shard index argument (0-22).
Shared	script / Slurm script	Pre-process data for the Shared scenario. Make sure to run this script with a shard index argument (0-7).
Baseline (Raw)	script / Slurm script	Pre-process data for the baseline (Raw). Make sure to run this script with a shard index argument (0-22).

Data for ablation studies

Please use the following scripts to pre-process the data for ablation studies:

Ablation Study	Script	Description
100% L2T Data	script / Slurm script	Pre-process data for the 100% L2T data ablation study. Make sure to run this script with a shard index argument (0-22).
75% L2T Data	script / Slurm script	Pre-process data for the 75% L2T data ablation study. Make sure to run this script with a shard index argument (0-22).
25% L2T Data	script / Slurm script	Pre-process data for the 25% L2T data ablation study. Make sure to run this script with a shard index argument (0-22).
Single task (Char Count)	script / Slurm script	Pre-process data for the single task (Char Count) ablation study. Make sure to run this script with a shard index argument (0-5).
Single task (Masked Char)	script / Slurm script	Pre-process data for the single task (Masked Char) ablation study. Make sure to run this script with a shard index argument (0-5).
Single task (Space)	script / Slurm script	Pre-process data for the single task (Space) ablation study. Make sure to run this script with a shard index argument (0-5).
Single task (Typo)	script / Slurm script	Pre-process data for the single task (Typo) ablation study. Make sure to run this script with a shard index argument (0-5).
Single task (Last)	script / Slurm script	Pre-process data for the single task (Last) ablation study. Make sure to run this script with a shard index argument (0-5).
Single task (Masked Word)	script / Slurm script	Pre-process data for the single task (Masked Word) ablation study. Make sure to run this script with a shard index argument (0-5).
Single task (Random Word)	script / Slurm script	Pre-process data for the single task (Random Word) ablation study. Make sure to run this script with a shard index argument (0-5).
Single task (Shuffle)	script / Slurm script	Pre-process data for the single task (Shuffle) ablation study. Make sure to run this script with a shard index argument (0-5).
Single task (Token Type)	script / Slurm script	Pre-process data for the single task (Token Type) ablation study. Make sure to run this script with a shard index argument (0-5).
Single task (Deletion)	script / Slurm script	Pre-process data for the single task (Deletion) ablation study. Make sure to run this script with a shard index argument (0-5).
Single task (Reordering)	script / Slurm script	Pre-process data for the single task (Reordering) ablation study. Make sure to run this script with a shard index argument (0-5).
Single task (Fill Middle)	script / Slurm script	Pre-process data for the single task (Fill Middle) ablation study. Make sure to run this script with a shard index argument (0-5).
Single task (Half)	script / Slurm script	Pre-process data for the single task (Half) ablation study. Make sure to run this script with a shard index argument (0-5).
Single task (One)	script / Slurm script	Pre-process data for the single task (One) ablation study. Make sure to run this script with a shard index argument (0-5).

Pre-training

Please use the following scripts to pre-train the models for different scenarios:

Scenario	Scale	Script	Description
Disjoint	1B	script / Slurm script	Pre-train 1B L2T model for the Disjoint scenario.
Disjoint	500M	script / Slurm script	Pre-train 500M L2T model for the Disjoint scenario.
Shared	1B	script / Slurm script	Pre-train 1B L2T model for the Shared scenario.
Shared	500M	script / Slurm script	Pre-train 500M L2T model for the Shared scenario.
Baseline (Raw; Disjoint)	1B	script / Slurm script	Pre-train 1B Raw model for the Baseline (Raw; Disjoint) scenario.
Baseline (Raw; Disjoint)	500M	script / Slurm script	Pre-train 500M Raw model for the Baseline (Raw; Disjoint) scenario.
Baseline (Raw; Shared)	1B	script / Slurm script	Pre-train 1B Raw model for the Baseline (Raw; Shared) scenario.
Baseline (Raw; Shared)	500M	script / Slurm script	Pre-train 500M Raw model for the Baseline (Raw; Shared) scenario.

Ablation studies

Please use the following scripts to pre-train the models for ablation studies:

Ablation Study	Scale	Script	Description
100% L2T Data	500M	script / Slurm script	Pre-train 500M L2T model for the 100% L2T data ablation study.
75% L2T Data	500M	script / Slurm script	Pre-train 500M L2T model for the 75% L2T data ablation study.
25% L2T Data	500M	script / Slurm script	Pre-train 500M L2T model for the 25% L2T data ablation study.
Single task (Char Count)	500M	script / Slurm script	Pre-train 500M L2T model for the single task (Char Count) ablation study.
Single task (Masked Char)	500M	script / Slurm script	Pre-train 500M L2T model for the single task (Masked Char) ablation study.
Single task (Space)	500M	script / Slurm script	Pre-train 500M L2T model for the single task (Space) ablation study.
Single task (Typo)	500M	script / Slurm script	Pre-train 500M L2T model for the single task (Typo) ablation study.
Single task (Last)	500M	script / Slurm script	Pre-train 500M L2T model for the single task (Last) ablation study.
Single task (Masked Word)	500M	script / Slurm script	Pre-train 500M L2T model for the single task (Masked Word) ablation study.
Single task (Random Word)	500M	script / Slurm script	Pre-train 500M L2T model for the single task (Random Word) ablation study.
Single task (Shuffle)	500M	script / Slurm script	Pre-train 500M L2T model for the single task (Shuffle) ablation study.
Single task (Token Type)	500M	script / Slurm script	Pre-train 500M L2T model for the single task (Token Type) ablation study.
Single task (Deletion)	500M	script / Slurm script	Pre-train 500M L2T model for the single task (Deletion) ablation study.
Single task (Reordering)	500M	script / Slurm script	Pre-train 500M L2T model for the single task (Reordering) ablation study.
Single task (Fill Middle)	500M	script / Slurm script	Pre-train 500M L2T model for the single task (Fill Middle) ablation study.
Single task (Half)	500M	script / Slurm script	Pre-train 500M L2T model for the single task (Half) ablation study.
Single task (One)	500M	script / Slurm script	Pre-train 500M L2T model for the single task (One) ablation study.

Evaluation

Linguistic competence and General benchmark evaluations

Please use the following scripts to evaluate the models for different scenarios:

L2T models: script / Slurm script
Baseline (Raw) models: script / Slurm script

Note

When running the evaluation scripts, please make sure to specify the correct model path as the first argument. For example, for evaluating the 1B L2T model trained in the Disjoint scenario, you would use:

sbatch /path/to/l2t/repository/evaluation/scripts/wrap_l2t.sh $SCRATCH/models/l2t-1b-disjoint

The corresponding evaluation results will be saved in /path/to/l2t/repository/evaluation/logs/l2t/ directory.

Psychometric evaluations

The psychometric evaluation is based on the EMNLP 2024 paper: Development of Cognitive Intelligence in Pre-trained Language Models by Raj Sanjay Shah, Khushi Bhardwaj, and Sashank Varma.

If you use the psychometric evaluation scripts, please make sure to cite the following paper:

@inproceedings{shah-etal-2024-development,
    title = "Development of Cognitive Intelligence in Pre-trained Language Models",
    author = "Shah, Raj Sanjay  and
      Bhardwaj, Khushi  and
      Varma, Sashank",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.emnlp-main.539/",
    doi = "10.18653/v1/2024.emnlp-main.539",
    pages = "9632--9657",
}

To streamline the evaluation process, we redesigned the original scripts and placed them in the evaluation/src/psychometric/ directory. Please refer to the README in that directory for detailed instructions on how to run the psychometric evaluations.

Model Checkpoints

We released the pre-trained model checkpoints on Hugging Face. Below is the list of available models:

Model	Scenario	Scale	Hugging Face Link
L2T	Disjoint	1B	link
L2T	Disjoint	500M	link
L2T	Shared	1B	link
L2T	Shared	500M	link
Baseline (Raw)	Disjoint	1B	link
Baseline (Raw)	Disjoint	500M	link
Baseline (Raw)	Shared	1B	link
Baseline (Raw)	Shared	500M	link
Ablation: 100% L2T Data		500M	link
Ablation: 75% L2T Data		500M	link
Ablation: 25% L2T Data		500M	link
Ablation: Single task (Char Count)		500M	link
Ablation: Single task (Masked Char)		500M	link
Ablation: Single task (Space)		500M	link
Ablation: Single task (Typo)		500M	link
Ablation: Single task (Last)		500M	link
Ablation: Single task (Masked Word)		500M	link
Ablation: Single task (Random)		500M	link
Ablation: Single task (Shuffle)		500M	link
Ablation: Single task (Token Type)		500M	link
Ablation: Single task (Deletion)		500M	link
Ablation: Single task (Reordering)		500M	link
Ablation: Single task (Fill Middle)		500M	link
Ablation: Single task (Half)		500M	link
Ablation: Single task (One)		500M	link

Pre-training Datasets

While we have already uploaded the datasets to Hugging Face, we are facing storage limitations that prevent us from making them publicly accessible at this time (i.e., we are over the storage limit for our Hugging Face account and cannot make the datasets public until we resolve this issue). Unfortunately, as we do not plan to increase our storage limit in the near future (which requires a paid subscription), we are unable to provide public access to the datasets at this time. Nonetheless, as we provide the scripts for pre-processing the data, you can use those scripts to generate the datasets on your end.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Citation

If you find this work useful in your research, please consider citing the following paper:

@inproceedings{yamaguchi-etal-2026-enhancing,
    title = "Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks",
    author = "Yamaguchi, Atsuki  and
      Mi, Maggie  and
      Aletras, Nikolaos",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 2: Short Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-short.27/",
    doi = "10.18653/v1/2026.acl-short.27",
    pages = "316--336",
    ISBN = "979-8-89176-391-3",
}

Name		Name	Last commit message	Last commit date
evaluation		evaluation
preprocessing		preprocessing
training		training
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
motivation.png		motivation.png

Navigation Menu

Repository files navigation

Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks

Installation

For AMD GPUs

For NVIDIA GPUs

Pre-processing

Data for ablation studies

Pre-training

Ablation studies

Evaluation

Linguistic competence and General benchmark evaluations

Psychometric evaluations

Model Checkpoints

Pre-training Datasets

License

Citation

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks

Installation

For AMD GPUs

For NVIDIA GPUs

Pre-processing

Data for ablation studies

Pre-training

Ablation studies

Evaluation

Linguistic competence and General benchmark evaluations

Psychometric evaluations

Model Checkpoints

Pre-training Datasets

License

Citation

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages