GPT-3
Summary: GPT-3 (Generative Pre-trained Transformer 3) — 175B parameter decoder-only Transformer. Demonstrated in-context learning (few-shot) without gradient updates. Validated scaling laws: performance scales smoothly with compute, params, data.
Details
For Models
- Architecture: Transformer Decoder (96 layers, 12288 hidden, 96 heads, 175B params)
- Training: 300B tokens (CommonCrawl filtered, WebText2, Books1/2, Wikipedia)
- Objective: Autoregressive LM (next token prediction)
- Key Finding: Few-shot learning — prompt with examples → model generalizes without weight updates
- API: OpenAI API (original); open replications: GPT-NeoX, BLOOM, LLaMA, Falcon
GPT Family
| Model | Year | Params | Key Contribution |
|---|---|---|---|
| GPT-1 | 2018 | 117M | Generative pretraining + discriminative fine-tuning |
| GPT-2 | 2019 | 1.5B | Scaling works; zero-shot task transfer |
| GPT-3 | 2020 | 175B | In-context few-shot learning; scaling laws |
| GPT-3.5 | 2022 | — | RLHF (InstructGPT → ChatGPT) |
| GPT-4 | 2023 | ~1T+ | Multimodal, stronger reasoning |
Significance
Paradigm shift: From "pretrain + fine-tune" (BERT) to "pretrain + prompt" (GPT-3).
For your research:
- LoRA/QLoRA origin: PEFT developed for GPT-scale models where full FT is impossible
- Scaling laws: Kaplan et al. (2020) — your consumer-GPU projects test efficient scaling
- Decoder-only dominance: GPT-3 cemented decoder-only for LLMs; ViT uses encoder but multimodal (Flamingo, BLIP-2) uses decoder
- Emergent abilities: Your DSA-ViT tests if sparse attention preserves emergence at smaller scale
Related
- transformer-architecture — Decoder stack
- lora-qloра — PEFT for GPT-scale
- google-research — Contrast: BERT (encoder) vs GPT (decoder)
Sources
- public-knowledge: Brown et al., "Language Models are Few-Shot Learners" (NeurIPS 2020)