Tensors and Dynamic neural networks in Python with strong GPU acceleration

PyTorch native quantization and sparsity for training and inference

A native PyTorch Library for large model training

Neptune Fetcher is designed to separate data retrieval capabilities from the regular neptune package. This separation makes data fetching more efficient and improves performance.

Enabling PyTorch on Google TPU

Logging module with nicer formatting

Pretrain, finetune, deploy 20+ LLMs on your own data. Uses state-of-the-art techniques: flash attention, FSDP, 4-bit, LoRA, and more.

Essential guides and programming tools in my toolbox (with focus on ML Training)

Blazingly fast, distributed streaming of training data from any cloud storage for training AI models

Source to source compiler for PyTorch. It makes PyTorch programs faster on single accelerators and distributed.

A Fusion Code Generator for NVIDIA GPUs (commonly known as "nvFuser")

A framework for few-shot evaluation of autoregressive language models.

NeurIPS Large Language Model Efficiency Challenge: 1 LLM + 1GPU + 1Day

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) precision on Hopper GPUs, to provide better performance with lower memory utilization in bot…

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Outlining techniques for improving the training performance of your PyTorch model without compromising its accuracy

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

FFCV: Fast Forward Computer Vision (and other ML workloads!)

Build and train PyTorch models and connect them to the ML lifecycle using Lightning App templates, without handling DIY infrastructure, cost management, scaling, and other headaches.

A latent text-to-image diffusion model

Taming Transformers for High-Resolution Image Synthesis

CPU & CUDA implementation of several neural network utils

Ongoing research training transformer models at scale

A deep learning toolkit for handwritten document analysis