In this work, we train small custom GPT on Moses and Guacamol dataset with next token prediction task. The model is then used for unconditional and conditional molecular generation. We compare our model with previous approaches on the Moses and Guacamol datasets. Saliency maps are obtained for interpretability using Ecco library.

• The processed Guacamol and MOSES datasets in csv format can be downloaded from this link:

https://drive.google.com/drive/folders/1LrtGru7Srj_62WMR4Zcfs7xJ3GZr9N4E?usp=sharing

https://www.kaggle.com/datasets/virajbagal/ligflow-datasets (working link)

• Original Guacamol dataset can be found here:

https://github.com/BenevolentAI/guacamol

• Original Moses dataset can be found here:

https://github.com/molecularsets/moses

• All trained weights can be found here:

https://www.kaggle.com/virajbagal/ligflow-final-weights (old weights, prior to modernization)

To train the model, make sure you have the datasets' csv file in the same directory as the code files.

./train_moses.sh

./train_guacamol.sh

./generate_guacamol_prop.sh

./generate_moses_prop_scaf.sh

If you find this work useful, please cite:

Bagal, Viraj; Aggarwal, Rishal; Vinod, P. K.; Priyakumar, U. Deva (2021): MolGPT: Molecular Generation using a Transformer-Decoder Model. ChemRxiv. Preprint. https://doi.org/10.26434/chemrxiv.14561901.v1

The transformer-decoder was modernized with three contemporary components and the effect of each was attributed via controlled, multi-seed ablations on MOSES and GuacaMol. Components are param-matched (modern ≈6.35M vs baseline ≈6.38M) and reach the same training loss (baseline 0.310–0.319, modern 0.309–0.316), so every difference below is an inductive-bias effect, not a capacity or fit difference.

• RoPE — replaced the learned absolute positional embedding (rotation on Q/K of SMILES tokens only; prepended condition tokens left unrotated).
• SwiGLU — bias-free down(SiLU(gate(x))·up(x)) MLP, hidden=682, param-matched.
• RMSNorm — replaced all three LayerNorms.

Headline: modernization is RoPE wearing a SwiGLU/RMSNorm coat. RoPE alone carries both the novelty gain and the FCD distribution-match penalty (a clean trade-off); SwiGLU and RMSNorm are behaviorally neutral, cost-free modernizations. Pick by use case — de-novo / novelty → modern; benchmark / FCD-matching → baseline. Full writeup, repro steps, and caveats in MODERNIZATION.md.

Create the conda env from environment.yml, then run the two post-install steps that can't be expressed declaratively (both are documented inline in the file — molsets needs --no-deps, and guacamol needs a one-line scipy patch):

conda env create -f environment.yml
conda activate molgpt

# (1) MOSES metrics — must be --no-deps (molsets would otherwise downgrade numpy/pandas/scipy)
pip install molsets==0.3.1 --no-deps

# (2) patch guacamol for modern scipy (scipy.histogram was removed; it equals numpy.histogram)
sed -i 's/from scipy import histogram/from numpy import histogram/' \
  "$(python -c 'import guacamol,os;print(os.path.dirname(guacamol.__file__))')/utils/chemistry.py"

The torch pin is the CUDA 12.6 wheel — edit the --extra-index-url/+cu126 tag in environment.yml for a different CUDA (or CPU). Datasets are not bundled: rebuild datasets/moses2.csv with python experiments/preprocess_moses.py, and fetch datasets/guacamol2.csv from Kaggle (virajbagal/ligflow-datasets).

All commands run from the inner molgpt/ code directory (the one containing train.py and experiments/), with the env active (conda activate molgpt). Checkpoints are written to ../cond_gpt/weights/. (The experiments/run_*.sh drivers cd to this directory automatically, so they can be launched from anywhere once the env is active.)

MOSES — architecture ablation (--arch {baseline,modern}):

python experiments/train_seeded.py --arch modern   --seed 1 --epochs 10 --run_name unconditional_moses_modern_s1
python experiments/train_seeded.py --arch baseline --seed 1 --epochs 10 --run_name unconditional_moses_baseline_s1

GuacaMol — per-component ablation (--config {baseline,rope,swiglu,rmsnorm,modern}):

# unconditional (single component on, or 'modern' for all three)
python experiments/train_ablate.py --config rope --seed 42 --epochs 10 --run_name guaca_uncond_rope_s42

# property-conditioned
python experiments/train_ablate.py --config modern --seed 42 --epochs 10 --run_name guaca_logp_modern_s42 \
    --num_props 1 --props logp

# scaffold-conditioned
python experiments/train_ablate.py --config modern --seed 42 --epochs 10 --run_name guaca_scaf_modern_s42 \
    --scaffold

# scaffold + property
python experiments/train_ablate.py --config modern --seed 42 --epochs 10 --run_name guaca_scaflogp_modern_s42 \
    --num_props 1 --props logp --scaffold

Evaluation (add --baseline for baseline-arch MOSES checkpoints; modern is the default):

# MOSES decoding-temperature frontier (fast metrics) -> datasets/sweep_<tag>.csv
python experiments/sweep_decode.py --ckpt ../cond_gpt/weights/unconditional_moses_modern_s1.pt \
    --temps "0.7 0.9 1.0 1.2 1.6" --out datasets/sweep_modern_s1.csv

# MOSES full metrics (FCD/SNN/Frag/Scaf/...) at one temperature -> datasets/moses_metrics_<tag>_T1.0.json
python experiments/gen_eval_moses.py --ckpt ../cond_gpt/weights/unconditional_moses_modern_s1.pt \
    --temp 1.0 --tag modern_s1

# GuacaMol (config + conditioning are auto-read from the checkpoint's sidecar .json)
python experiments/eval_guaca.py --run_name guaca_uncond_rope_s42 --temp 1.0

End-to-end, idempotent pipelines (each phase skips already-finished work, so re-running is safe):

# MOSES multi-seed: train seeds 1,2 (both archs) + 25-epoch modern -> eval all -> aggregate
bash experiments/run_train_all.sh
bash experiments/run_eval_all.sh
python experiments/aggregate_seeds.py        # -> experiments/multiseed_results.md

# GuacaMol full matrix: train -> eval -> aggregate
bash experiments/run_guaca_all.sh            # -> experiments/guaca_ablation_results.md

To widen the GuacaMol matrix, edit A_SEEDS / B_SEEDS / B_MODES in experiments/run_train_guaca.sh and re-run run_guaca_all.sh.

Each component on (✓) / off (✗). Validity/Unique/Novelty/KL-div: higher = better. FCD: lower = better (distance to held-out reference). The trade-off isolates to RoPE alone (FCD 0.992→1.253, ≈10σ, zero seed overlap); SwiGLU/RMSNorm are FCD-neutral.

The FCD trade-off reproduces on a second benchmark: modern is ~10σ worse on FCD/Test (0.571→0.906) while fast quality metrics stay identical.

The validity "gap" is a decoding-temperature artifact, not architectural — both archs are ≥0.978 valid at T≤1.2 and collapse to ~0.85 at T=1.6. Report at T=1.0–1.2.

Modern is significantly more novel at low temperature (all 3 per-seed deltas positive); the edge washes out by T≥1.2. With n=3 the t-test is low-power, so sign-consistency is the primary evidence.

Where the unconditional story is a trade-off, conditional generation is neutral-to-better for modernization. MAD = mean abs. deviation of generated property from target (lower = better); scaffold-match = fraction matching the conditioning scaffold.

More epochs → validity ↑ but novelty ↓ with FCD unchanged, i.e. it moves away from the paper's joint numbers. The residual gap vs the manuscript is data/preprocessing/reporting, not training budget; the FCD gap is genuinely architectural.