WinQ Accelerating Quantization-Aware Training for LLMs around Saddle Points

FedComLoc: Communication-Efficient Distributed Training of Sparse and Quantized Models. TMLR, 2025.

[ICML24] Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for LLMs

This repo is meant to serve as a guide for Machine Learning/AI technical interviews.

Summaries and resources for Designing Machine Learning Systems book (Chip Huyen, O'Reilly 2022)

Curated coding interview preparation materials for busy software engineers

最完整的AI算法面试题目仓库，1000道，25个类目

Implementation of the paper: "FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneity" (ICLR 2024)

Schedule-Free Optimization in PyTorch

Official Pytorch repository for Extreme Compression of Large Language Models via Additive Quantization https://arxiv.org/pdf/2401.06118.pdf and PV-Tuning: Beyond Straight-Through Estimation for Ext…

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

Implementation of "BitNet: Scaling 1-bit Transformers for Large Language Models" in pytorch

Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

EF-BV: A Unified Theory of Error Feedback and Variance Reduction Mechanisms for Biased and Unbiased Compression in Distributed Optimization. NeurIPS, 2022

Variance Reduced ProxSkip: Algorithm, Theory and Application to Federated Learning. NeurIPS, 2022

Examples in the MLX framework

The TinyLlama project is an open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens.

S-LoRA: Serving Thousands of Concurrent LoRA Adapters

⛷️ LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training (EMNLP 2024)

Instruct-tune LLaMA on consumer hardware

An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.

Code for loralib, an implementation of "LoRA: Low-Rank Adaptation of Large Language Models"

[CVPR 2023] DepGraph: Towards Any Structural Pruning; LLMs, Vision Foundation Models, etc.

[NeurIPS 2023] LLM-Pruner: On the Structural Pruning of Large Language Models. Support Llama-3/3.1, Llama-2, LLaMA, BLOOM, Vicuna, Baichuan, TinyLlama, etc.

The official GitHub page for the survey paper "A Survey of Large Language Models".

A flexible Federated Learning Framework based on PyTorch, simplifying your Federated Learning research.

ICLR2023 statistics

Build terminal dashboards using ascii/ansi art and javascript