A LLaMA2-7b chatbot with memory running on CPU, and optimized using smooth quantization, 4-bit quantization or Intel® Extension For PyTorch with bfloat16.

mxnet GluonCV quantization binary ternary models

Convert a Node-API value representing a two-dimensional strided array to a signed 8-bit integer array.

Size (in bytes) of an 8-bit signed integer.

Convert a Node-API value to a signed 8-bit integer array.

INT8 calibrator for ONNX model with dynamic batch_size at the input and NMS module at the output. C++ Implementation.

Practicing C data types using the sizeof function

development quantization framework

MT-Yolov6 TensorRT Inference with Python.

VB.NET api wrapper for llm-inference chatllm.cpp

C# api wrapper for llm-inference chatllm.cpp

Maximum signed 8-bit integer.

8-bit signed integer mathematical constants.

Test if a value is an Int8Array.

Convert a Node-API value representing a strided array to a signed 8-bit integer array.

Int8Array.

Minimum signed 8-bit integer.

LLM-Lora-PEFT_accumulate explores optimizations for Large Language Models (LLMs) using PEFT, LORA, and QLORA. Contribute experiments and implementations to enhance LLM efficiency. Join discussions and push the boundaries of LLM optimization. Let's make LLMs more efficient together!