An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library.

Repo for ICLR22 "Capturing Structural Locality in Non-parametric Language Models"

Code for the model presented in the paper: "code2seq: Generating Sequences from Structured Representations of Code"

Code for CodeT5: a new code-aware pre-trained encoder-decoder model.