HuggingFace Transformers

Summary: The de facto standard Python library for Transformer models. Provides model architectures, training utilities (Trainer), PEFT (LoRA/QLoRA), TRL (RLHF), and seamless HuggingFace Hub integration.

Details

  • Type: Open-source ML library / ecosystem
  • Language: Python (PyTorch, TensorFlow, JAX backends)
  • Key Components:
    • Models: 1000+ architectures (BERT, GPT, T5, LLaMA, ViT, Whisper, etc.)
    • Trainer: High-level training loop with distributed, mixed precision, callbacks
    • PEFT: Parameter-efficient fine-tuning (LoRA, QLoRA, AdaLoRA, IA3, Prompt Tuning)
    • TRL: Transformer Reinforcement Learning (SFT, DPO, PPO, Reward Modeling)
    • Accelerate: Multi-GPU/TPU training with minimal code changes
    • Optimum: Export to ONNX, TensorRT, OpenVINO for deployment
  • Hub Integration: model.push_to_hub(), AutoModel.from_pretrained("user/model")
  • Website: huggingface.co/docs/transformers

For Your Workflow

Task HF Tool Your Wiki
Load pretrained ViT ViTModel.from_pretrained("google/vit-base-patch16-224") transformer-architecture
LoRA fine-tuning peft.LoraConfig, get_peft_model() lora-qloра
QLoRA 4-bit bitsandbytes + LoraConfig(quantization_config=bnb_4bit) lora-qloра
FlashAttention attn_implementation="flash_attention_2" flash-attention
Distributed training Accelerator(), Trainer —
Deploy to Hub push_to_hub(), model cards —

Code Snippets

LoRA + FlashAttention + Trainer:

from transformers import Trainer, TrainingArguments, AutoModelForCausalLM
from peft import LoraConfig, get_peft_model
import torch

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    attn_implementation="flash_attention_2",
    torch_dtype=torch.bfloat16,
)
model = get_peft_model(model, LoraConfig(r=16, lora_alpha=32, target_modules="all-linear"))
trainer = Trainer(model=model, args=TrainingArguments(...))
trainer.train()

Related

Sources

  • public-knowledge: HF Transformers documentation