LoRA & QLoRA

Summary: LoRA (Low-Rank Adaptation) freezes pretrained weights and adds trainable low-rank matrices A ∈ R^{r × d}, B ∈ R^{d × r} to each layer. QLoRA (Quantized LoRA) combines 4-bit NF4 quantization with LoRA, enabling billion-parameter model fine-tuning on consumer GPUs.

LoRA (Low-Rank Adaptation)

Core Hypothesis

In plain English: During fine-tuning, weight updates Delta W have low intrinsic rank. Instead of learning full d × d updates, learn a low-rank decomposition B A where r ll d.

Mathematical Formulation

For a pretrained weight matrix W_0 ∈ R^{d × d}:

W = W_0 + Delta W = W_0 + B A

where:

  • A ∈ R^{r × d} (down-projection)
  • B ∈ R^{d × r} (up-projection)
  • r ll d (typically r = 8, 16, 32, 64)
  • W_0 frozen, only A, B trained

Forward pass:

h = W_0 x + B(A x) = W_0 x + Delta W x

Initialization: A ~ N(0, Sigma^2), B = 0 → Delta W = 0 at start (preserves pretrained behavior)

Parameter Efficiency

Model Full FT Params LoRA (r=8) Params Reduction
GPT-3 175B 175B 4.7M 37,000×
LLaMA 7B 7B 4.2M 1,600×
BERT-large 340M 1.1M 300×

Where to Apply LoRA

Original paper (Hu et al., 2021): Only attention projection matrices (W_q, W_k, W_v, W_o)

Later findings (He et al., 2022; Liu et al., 2024): Apply to all linear layers including FFN (W_{up}, W_{down}, W_{gate}) for better performance.

Scaling Factor alpha

h = W_0 x + (alpha)/(r) B(A x)
  • alpha typically = r or 2r (e.g., r=16, alpha=32)
  • Equivalent to learning rate scaling for A,B

Dropout

Apply dropout to A x (the low-rank intermediate) during training.

QLoRA (Quantized LoRA)

4-bit NormalFloat (NF4)

Quantization scheme optimized for normally distributed weights:

  1. Block-wise quantization: Partition weights into blocks of 64 elements
  2. Per-block quantization constant: Learn optimal scale per block (33-bit)
  3. NF4 data type: 4-bit values mapped to theoretical quantiles of N(0,1)

NF4 precision mapping (theoretical quantiles):

Binary Value Binary Value
0000 -1.0 1000 0.0
0001 -0.696 1001 0.191
0010 -0.525 1010 0.323
0011 -0.395 1011 0.441
0100 -0.281 1100 0.596
0101 -0.180 1101 0.752
0110 -0.091 1110 1.0
0111 0.0 1111 No value

Double Quantization

  • First quantization: Weights → 4-bit NF4 + 32-bit absmax per block
  • Second quantization: Absmax constants → 8-bit float (further 0.5 bits/param)

Paged Optimizers

  • Offload optimizer states (Adam moments) to CPU RAM
  • Page back to GPU only when needed
  • Prevents OOM during large model training

QLoRA Memory Footprint (from paper)

Model Params 16-bit FT QLoRA (4-bit) Reduction
LLaMA-7B 7B >24GB 6GB 4×
LLaMA-13B 13B >40GB 10GB 4×
LLaMA-33B 33B >80GB 24GB 3.3×
LLaMA-65B 65B >160GB 48GB 3.3×

Enables: 65B fine-tuning on single 48GB GPU (A100 80GB) or dual 24GB (RTX 3090)

LoRA Variants & Extensions

Variant Key Idea Reference
AdaLoRA Adaptive rank allocation via SVD Zhang et al. (2023)
LoRA+ Different LR for A vs B Liu et al. (2024)
DoRA Decompose magnitude + direction Liu et al. (2024)
PiSSA Principal singular values init Meng et al. (2024)
VeRA Shared frozen A, learnable B + scaling Krülle et al. (2024)
LoRA-GA Gradient accumulation on A —

Practical Guidelines

Rank Selection

Task Type Recommended r Notes
Classification/Simple QA 4-8 Low rank sufficient
Generation/Complex reasoning 16-64 Higher capacity needed
Multi-task 32-128 More capacity for diversity
Full parameter tuning recovery 256+ Approaches full FT

Target Modules

# Minimal (attention only)
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj"]

# Recommended (all linear)
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", 
                  "gate_proj", "up_proj", "down_proj"]

Hyperparameters

lora_config = LoraConfig(
    r=16,                    # Rank
    lora_alpha=32,           # Scaling (often 2*r)
    lora_dropout=0.05,       # Dropout
    target_modules=target_modules,
    bias="none",             # Or "lora_only"
    task_type="CAUSAL_LM",
)

LoRA + FlashAttention

  • Compatible: FlashAttention computes attention; LoRA modifies projections
  • Combined enables long-context fine-tuning on consumer GPUs
  • Sequence length 16K-32K feasible on 24GB GPU with QLoRA + FlashAttention-2

Limitations

  • Inference latency: Two extra matmuls per layer (but mergeable: W_{merged} = W_0 + BA)
  • Composition: Multiple LoRAs not trivially combinable
  • Capacity: Very low rank may underfit complex tasks
  • Distribution shift: Large domain shift may need higher rank or full FT

Related Concepts

Sources

  • arxiv-2106.09685: Sections 3, 4, 5 — LoRA formulation, experiments, ablation
  • arxiv-2305.14314: Sections 3, 4 — QLoRA 4-bit NF4, double quantization, paged optimizer