Utilities for efficient fine-tuning, inference and evaluation of code generation models
-
Updated
Oct 3, 2023 - Python
#
flash-attention
Here are 55 public repositories matching this topic...
Python package for rematerialization-aware gradient checkpointing
-
Updated
Oct 31, 2023 - Python
Train llm (bloom, llama, baichuan2-7b, chatglm3-6b) with deepspeed pipeline mode. Faster than zero/zero++/fsdp.
nlp bloom pipeline pytorch deepspeed llm full-finetune model-parallization flash-attention llama2 baichuan2-7b chatglm3-6b mixtral-8x7b
-
Updated
Feb 5, 2024 - Python
Triton implementation of FlashAttention2 that adds Custom Masks.
deep-learning triton attention cuda-kernels attention-mechanism triton-lang flash-attention flash-attention-2
-
Updated
Aug 14, 2024 - Python
Toy Flash Attention implementation in torch
-
Updated
Sep 22, 2024 - Python
Long term project about a custom AI architecture. Consist of cutting-edge technique in machine learning such as Flash-Attention, Group-Query-Attention, ZeRO-Infinity, BitNet, etc.
python machine-learning deep-learning jupyter-notebook pytorch transformer bitnet pytorch-lightning vision-transformer large-language-models low-rank-adaptation flash-attention
-
Updated
Oct 15, 2024 - Python
Fast and memory efficient PyTorch implementation of the Perceiver with FlashAttention.
-
Updated
Nov 4, 2024 - Python
Decoder-only LLM trained on the Harry Potter books.
transformer byte-pair-encoding rotary-position-embedding large-language-model flash-attention grouped-query-attention
-
Updated
Dec 20, 2024 - Python
Training GPT-2 on FineWeb-Edu in JAX/Flax
-
Updated
Dec 28, 2024 - Python
[CVPR 2025 Highlight] The official CLIP training codebase of Inf-CL: "Breaking the Memory Barrier: Near Infinite Batch Size Scaling for Contrastive Loss". A super memory-efficiency CLIP training scheme.
-
Updated
Jan 16, 2025 - Python
Building Native Sparse Attention
-
Updated
Feb 20, 2025 - Python
A flexible and efficient implementation of Flash Attention 2.0 for JAX, supporting multiple backends (GPU/TPU/CPU) and platforms (Triton/Pallas/JAX).
-
Updated
Mar 4, 2025 - Python
MoBA: Mixture of Block Attention for Long-Context LLMs
-
Updated
Apr 3, 2025 - Python
Grouped-Tied Attention by Zadouri, Strauss, Dao (2025).
transformers attention in-context-learning flash-attention in-context-reinforcement-learning algorithm-distillation
-
Updated
Jul 4, 2025 - Python
A from-scratch implementation of a T5 model modified with Rotary Position Embeddings (RoPE). This project includes the code for pre-training on the C4 dataset in streaming mode with Flash Attention 2.
nlp pytorch sequence-to-sequence language-model from-scratch rope pre-training huggingface t5 evaluation-benchmark llm rotary-position-embedding flash-attention c4-dataset span-corruption
-
Updated
Jul 9, 2025 - Python
Ring sliding window attention implementation with flash attention
-
Updated
Jul 25, 2025 - Python
Implementing modern DL systems from scratch — Transformers, Diffusion, Multimodal LLMs, FlashAttention, RLHF.
machine-learning ai deep-learning pytorch transformer from-scratch attention-is-all-you-need rag diffusion-models llm rlhf vision-language-model multimodal-large-language-models flash-attention
-
Updated
Jul 31, 2025 - Python
InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.
pytorch multi-modal gemma pipeline-parallelism transformers-models tensor-parallelism llava llm-training internlm flash-attention zero3 llm-framework sequence-parallelism internlm2 ring-attention deepspeed-ulysses llama3 910b
-
Updated
Aug 21, 2025 - Python
Improve this page
Add a description, image, and links to the flash-attention topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the flash-attention topic, visit your repo's landing page and select "manage topics."