(For educational purposes) generative model for bioRxiv titles.
Roadmap
- Implement attention https://arxiv.org/abs/1706.03762
- Load data from https://huggingface.co/datasets/laion/biorXiv_metadata
- tokenize data
- Pytorch data loader
- Token embedder and position embedder
- generate mode
- Implement train step and eval step (Overfit on one batch with a small model)
- Train small v0 model on CPU
- Scale up model and train v1 on Colab GPU