Quantization-Aware Training (QAT) & Post-Training Quantization (PTQ)

Summary: Quantization reduces model precision from FP16/BF16 to INT8/INT4, shrinking memory 2-4× and accelerating inference. QAT simulates quantization during training to preserve accuracy; PTQ quantizes after training with calibration data. QLoRA combines 4-bit PTQ with LoRA fine-tuning.

Quantization Basics

In plain English: Neural network weights are typically 16-bit floats (FP16). Quantization maps them to 8-bit or 4-bit integers: q = round(w / scale + zero_point). This makes models 2-4× smaller and faster on integer-optimized hardware.

Quantization Types

Aspect PTQ (Post-Training) QAT (Quantization-Aware Training)
When After full-precision training During training
Data needed Small calibration set (128-1024 samples) Full training data
Accuracy drop 1-5% (larger for INT4) <1% (often recovers)
Complexity Simple Requires fake-quant ops
Time Minutes Full training time

Symmetric vs Asymmetric

Symmetric (INT8, INT4):

q = clip(round ≤ ft((w)/(s)), -2^{b-1}, 2^{b-1}-1right)
  • Zero-point = 0
  • Used for weights (GPTQ, AWQ, NF4)

Asymmetric (UINT8):

q = clip(round ≤ ft((w)/(s) + z), 0, 2^b-1
  • Zero-point z ≠ 0
  • Used for activations (inputs/outputs)

PTQ Methods

1. GPTQ (Gradient-based PTQ) — Frantar et al., 2023

  • Per-layer optimization: Minimize ||W - hat{W}||^2 via Hessian-weighted error
  • Algorithm: Iterative quantization using inverse Hessian diagonal
  • Supports: INT3, INT4, INT8
  • Slow but high accuracy; ~hours for LLaMA-7B

2. AWQ (Activation-aware Weight Quantization) — Lin et al., 2023

  • Key insight: Not all weights equally important; protect weights for salient activations
  • Metric: S = abs(W) × abs(X)_{mean} — weight × activation magnitude
  • Per-channel scaling: Different scale per output channel
  • Fast: No Hessian, just calibration data pass

3. RTN (Round-to-Nearest) — Baseline

  • Simple per-tensor or per-channel scaling
  • Poor for INT4 (<50% accuracy on LLMs)

QAT Methods

Fake Quantization (Straight-Through Estimator)

def fake_quant(x, scale, zero_point, bits=8):
    # Forward: quantize + dequantize
    x_q = torch.round(x / scale + zero_point).clamp(qmin, qmax)
    x_dq = (x_q - zero_point) * scale
    return x_dq

# Backward: gradient passes through unchanged (STE)
# x.grad = x_dq.grad

Training Flow

  1. Start from pretrained FP16 model
  2. Insert fake-quant modules after each weight/activation
  3. Fine-tune with quantization simulation
  4. Export quantized weights (remove fake-quant)

Learned Quantization Parameters

  • Scales: Learnable per-channel/per-tensor
  • Zero-points: Learnable (asymmetric)
  • Bit-width: Can be mixed-precision (sensitive layers → higher bits)

4-bit Quantization: NF4 (NormalFloat)

QLoRA's 4-bit NF4 (Dettmers et al., 2023)

  • Block-wise quantization: 64-element blocks
  • NF4 datatype: Optimized for normal distribution N(0,1)
  • Double quantization: Quantize quantization constants (saves 0.5 bits/param)

NF4 Values (16 levels)

Level:  0    1    2    3    4    5    6    7    8    9   10   11   12   13   14   15
Value: -1.0 -0.7 -0.5 -0.4 -0.3 -0.2 -0.1  0.0  0.1  0.2  0.3  0.4  0.5  0.6  0.7  1.0

(Approximate quantiles of standard normal)

Memory Savings

Model FP16 4-bit NF4 Reduction
LLaMA-7B 13.5 GB 3.5 GB 3.9×
LLaMA-13B 25 GB 6.5 GB 3.8×
LLaMA-70B 135 GB 35 GB 3.9×

Mixed Precision Strategies

W8A8 (Weights INT8, Activations INT8)

  • Both weights and activations quantized
  • Requires QAT or advanced PTQ for accuracy
  • Tensor Cores: DP4A/DOT2 (INT8) → 2× FP16 throughput

W4A16 (Weights INT4, Activations FP16)

  • Common for LLM inference (weights bottleneck memory)
  • Dequantize weights on-the-fly: W_{deq} = (W_q - z) × s
  • Used by: QLoRA, GPTQ-4bit, AWQ-4bit

W4A4 (Both INT4)

  • Maximum compression
  • Significant accuracy drop without QAT
  • Emerging: QAT for W4A4 (e.g., LLM-QAT, ZeroQuant-V2)

Hardware Support

Hardware INT8 INT4 FP8
Ampere (A100, RTX 30) ✅ DP4A ❌ ❌
Hopper (H100) ✅ DP4A ✅ DP4A (via MMA) ✅ Tensor Cores
Ada Lovelace (RTX 40) ✅ DP4A ✅ DP4A ✅ Tensor Cores
Blackwell (B100/RTX 50) ✅ ✅ Native ✅ Native

Consumer GPU note: RTX 3090/4090 lack native INT4 tensor cores — use bitsandbytes CUDA kernels (dequant on-the-fly).

Accuracy Impact (Typical)

Task FP16 W8A8 (PTQ) W4A16 (PTQ) W4A16 (QAT)
Perplexity (WikiText2) 5.2 5.4 (+4%) 6.1 (+17%) 5.3 (+2%)
MMLU (7B) 45% 44% 41% 44%
GSM8K (7B) 30% 29% 25% 29%

Deployment Libraries

Library Method Backend Notes
bitsandbytes PTQ (8/4-bit) + QLoRA CUDA Research standard
GPTQ-for-LLaMA PTQ (GPTQ) CUDA Fast inference
AutoAWQ PTQ (AWQ) CUDA Fast, accurate
llama.cpp / GGML PTQ (k-quant) CPU/Apple Silicon Edge deployment
TensorRT-LLM PTQ/QAT TensorRT Production NVIDIA
ONNX Runtime PTQ/QAT ONNX Cross-platform

Best Practices

  1. Start with W4A16 PTQ (AWQ/GPTQ) — fastest path
  2. If accuracy drop >2%: Try QAT with learned scales
  3. Mixed precision: Keep sensitive layers (embedding, output, attention) at INT8
  4. Calibration data: Use 512-1024 diverse samples from target domain
  5. Evaluate end-to-end: Perplexity ≠ downstream task accuracy

Related Concepts

Sources

  • arxiv-2305.14314: QLoRA paper (NF4, double quantization, paged optimizer)
  • Frantar et al., "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers" (2023)
  • Lin et al., "AWQ: Activation-aware Weight Quantization for LLM Compression" (2023)