Most teams that want a fine-tuned language model don't have a GPU cluster — they have one workstation, or a single rented instance, and a deadline. This post covers the setup that has worked reliably across several client engagements for getting a 7B-parameter model fine-tuned on a single consumer GPU without sacrificing much quality.

Why quantization first

Full fine-tuning of a 7B model in fp16 needs roughly 60–80GB of GPU memory once you account for optimizer states, gradients, and activations. Almost no consumer card has that. QLoRA changes the arithmetic: the base model is loaded in 4-bit precision and frozen, and only small low-rank adapter matrices are trained on top of it. That drops the memory footprint enough to fit on a 24GB card, sometimes less, depending on sequence length and batch size.

The core trade being made is memory for a small amount of quantization noise. In practice, with double quantization and a properly chosen bnb_4bit_compute_dtype, the quality gap versus full fine-tuning is small enough to be worth it for the overwhelming majority of production use cases.

The setup that works

A few choices matter more than people expect:

  • Rank and alpha. Starting at rank 16 with alpha set to 2× the rank is a reasonable default. Going higher rarely buys much unless the target domain is very far from the base model's pretraining distribution.
  • Target modules. Applying LoRA to only the attention projection matrices is common, but including the MLP up/down projections tends to give a noticeable quality bump for a modest memory cost.
  • Learning rate. LoRA adapters tolerate a higher learning rate than full fine-tuning — something in the 1e-4 to 2e-4 range is a reasonable starting point, cosine-annealed over the run.
  • Gradient checkpointing. Turn this on. It roughly halves activation memory in exchange for a manageable amount of extra compute, and on a memory-constrained single GPU that trade is close to always worth it.

Where this breaks down

QLoRA is not a universal answer. If the target task requires the model to acquire genuinely new knowledge rather than adjust behavior — a new language with very different morphology, for instance — a frozen 4-bit base can be a real ceiling. That was part of the motivation behind FaseehGPT: for Arabic text generation, morphological understanding benefited from architecture-level changes, not just adapter tuning on top of an English-centric base model.

The other place this breaks down is inference. An adapter-based model still needs the adapters merged or the base model kept in memory alongside them, and 4-bit inference has its own throughput trade-offs. Fine-tuning cheaply and serving cheaply are different problems.

Practical checklist

  1. Quantize the base model in 4-bit with double quantization enabled.
  2. Apply LoRA to attention and MLP projections at rank 16, alpha 32, as a starting point.
  3. Enable gradient checkpointing.
  4. Use a cosine learning-rate schedule with warmup.
  5. Evaluate on a held-out set that reflects the actual deployment distribution, not just a generic benchmark — this is where most fine-tuning projects actually go wrong.

The tooling around this has matured a lot; see the QuantLLM project for a lightweight toolkit that wraps steps 1–3 into a smaller surface area.