Transformers documentation

HQQ

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

HQQ

Half-Quadratic Quantization (HQQ) optimizes its quantization parameters from the weights alone, so it doesn’t need calibration data. Use it to quantize a model to as few as 1 or 2 bits while it loads.

The table below summarizes what HQQ supports in Transformers.

ConfigHqqConfig
quant_methodhqq
WorkflowQuantize while loading
Bits1, 2, 3, 4, 8
HardwareCUDA, XPU
Save✅
Fine-tune✅
Auto-compile❌

Install HQQ

Install hqq 0.2.1 or later.

pip install -U hqq

Quantize a model with HQQ

Create a HqqConfig and pass it to from_pretrained(). The example below quantizes every linear layer to 4 bits in groups of 64 weights.

import torch
from transformers import AutoModelForCausalLM, HqqConfig

quantization_config = HqqConfig(nbits=4, group_size=64)
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-0.6B",
    quantization_config=quantization_config,
    device_map="cuda",
)

Configure HQQ

These HqqConfig parameters change how the weights are quantized.

ParameterDefaultEffect
nbits4Number of bits per weight. Supports 8, 4, 3, 2, and 1.
group_size64Number of weights that share a scale and zero point. Smaller groups are more accurate and use more memory.
axis1Axis to group weights along, either 0 or 1. The faster torchao and BitBlas backends work only with 1.
view_as_floatFalseStores the quantized weights as a float dtype, for distributed training.
skip_modules["lm_head"]Layer types to leave unquantized.

Quantize layer types with different settings

Pass dynamic_config to quantize each type of linear layer with its own settings. Each key is a layer tag, the module name with model, layers, and the layer number removed. For example, model.layers.3.self_attn.q_proj has the tag self_attn.q_proj.

The example below quantizes the attention projections to 4 bits and the MLP projections to 3 bits.

q4_config = {"nbits": 4, "group_size": 64}
q3_config = {"nbits": 3, "group_size": 32}
quantization_config = HqqConfig(dynamic_config={
    "self_attn.q_proj": q4_config,
    "self_attn.k_proj": q4_config,
    "self_attn.v_proj": q4_config,
    "self_attn.o_proj": q4_config,
    "mlp.gate_proj": q3_config,
    "mlp.up_proj": q3_config,
    "mlp.down_proj": q3_config,
})

A tag applies to that layer type in every decoder layer, so you can’t give one decoder layer different settings from the others. Linear layers whose tags aren’t in dynamic_config stay unquantized. The same tags are used to match skip_modules.

Choose a backend

HQQ runs the quantized layers through a backend from the hqq library. The PyTorch and CUDA backends work with PEFT fine-tuning and older GPUs. Switch backends with HQQLinear.set_backend.

from hqq.core.quantize import HQQBackend, HQQLinear

HQQLinear.set_backend(HQQBackend.PYTORCH)

For faster 4-bit inference, convert the quantized model to a fused kernel backend, like torchao, after loading it. The torchao backend needs axis=1, which is the default.

from hqq.utils.patching import prepare_for_inference

prepare_for_inference(model, backend="torchao_int4")

See the hqq Backend guide for the available backends.

Limitations

  • HQQ can’t offload to the CPU or disk. Loading with a device_map that includes "cpu" or "disk" raises a ValueError.
  • Layer tags can’t target one decoder layer, so dynamic_config and skip_modules apply to a layer type across the whole model.

Resources

Update on GitHub