You can also use the GGUF-my-repo space on Hugging Face to build your own quants without any setup.

Note: It is synced from llama.cpp main every 6 hours.

Example usage:

# obtain the official LLaMA model weights and place them in ./models
ls ./models
llama-2-7b tokenizer_checklist.chk tokenizer.model
# [Optional] for models using BPE tokenizers
ls ./models
<folder containing weights and tokenizer json> vocab.json
# [Optional] for PyTorch .bin models like Mistral-7B
ls ./models
<folder containing weights and tokenizer json>

# install Python dependencies
python3 -m pip install -r requirements.txt

# convert the model to ggml FP16 format
python3 convert_hf_to_gguf.py models/mymodel/

# quantize the model to 4-bits (using Q4_K_M method)
./llama-quantize ./models/mymodel/ggml-model-f16.gguf ./models/mymodel/ggml-model-Q4_K_M.gguf Q4_K_M

# update the gguf filetype to current version if older version is now unsupported
./llama-quantize ./models/mymodel/ggml-model-Q4_K_M.gguf ./models/mymodel/ggml-model-Q4_K_M-v2.gguf COPY

Run the quantized model:

# start inference on a gguf model
./llama-cli -m ./models/mymodel/ggml-model-Q4_K_M.gguf -cnv -p "You are a helpful assistant"

When running the larger models, make sure you have enough disk space to store all the intermediate files.

As the models are currently fully loaded into memory, you will need adequate disk space to save them and sufficient RAM to load them. At the moment, memory and disk requirements are the same.

Several quantization methods are supported. They differ in the resulting model disk size and inference speed.

The quantization formats Q4_0_4_4, Q4_0_4_8 and Q4_0_8_8 are block interleaved variants of the Q4_0 format, providing a data layout that is better suited for specific implementations of optimized mulmat kernels. Since these formats differ only in data layout, they have the same quantized size as the Q4_0 format.

(outdated)

• k-quants
• recent k-quants improvements and new i-quants
  • #2707
  • #2807
  • #4773 - 2-bit i-quants (inference)
  • #4856 - 2-bit i-quants (inference)
  • #4861 - importance matrix
  • #4872 - MoE models
  • #4897 - 2-bit quantization
  • #4930 - imatrix for all k-quants
  • #4951 - imatrix on the GPU
  • #4969 - imatrix for legacy quants
  • #4996 - k-qunats tuning
  • #5060 - Q3_K_XS
  • #5196 - 3-bit i-quants
  • quantization tuning, another one, and another one

Llama 2 7B

Llama 2 13B

Llama 2 70B