Attention Is All You Need — Paper Breakdown
TL;DR: The Transformer replaces RNNs/CNNs with pure attention. Encoder-decoder stacks of self-attention + FFN achieve SOTA translation (28.4 BLEU EN-DE, 41.8 EN-FR) at fraction of training cost. Introduces multi-head attention, scaled dot-product attention, sinusoidal positional encoding.
Paper Metadata
| Field | Value |
|---|---|
| Paper ID | arxiv-1706.03762 |
| Title | Attention Is All You Need |
| Authors | Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin |
| Venue/Year | NeurIPS 2017 |
| ArXiv/DOI | https://arxiv.org/abs/1706.03762 |
| Code | https://github.com/tensorflow/tensor2tensor |
Problem Statement
Prior dominant approaches: RNNs (LSTM/GRU), CNNs (ByteNet, ConvS2S) for seq2seq. Limitations:
- Sequential computation — RNNs require
O(n)sequential steps, prevent parallelization - Long-range dependencies — Path length
O(n)in RNNs,O(log_k n)in CNNs → hard to learn - Memory bottleneck — RNN hidden state compresses all history into fixed vector
Transformer solution: Self-attention connects all positions in O(1) sequential operations, path length O(1).
Method Overview
High-Level Idea (In Your Own Words)
The Transformer processes the entire sequence in parallel by computing attention between every token pair. Instead of reading tokens one-by-one (RNN) or with fixed windows (CNN), each token directly "looks at" all other tokens and decides how much to incorporate from each. Stack this 6 times → deep bidirectional context.
Architecture Diagram
Diagram: (Mermaid diagram - view source for diagram code)
graph TD Input[Input Tokens] --> Emb[Token Embedding + PosEncoding] Emb --> Enc[Encoder × 6] Enc --> Dec[Decoder × 6] Dec --> Out[Output Projection + Softmax]
subgraph Encoder Layer
E1[Multi-Head Self-Attention] --> E2[Add & Norm]
E2 --> E3[Position-wise FFN] --> E4[Add & Norm]
end
subgraph Decoder Layer
D1[Masked Multi-Head Self-Attn] --> D2[Add & Norm]
D2 --> D3[Encoder-Decoder Attention] --> D4[Add & Norm]
D4 --> D5[Position-wise FFN] --> D6[Add & Norm]
end
Key Components
| Component | Description | Novelty |
|---|---|---|
| Scaled Dot-Product Attention | softmax(QK^T/sqrt(d_k))V |
Scaling factor 1/sqrt(d_k) |
| Multi-Head Attention | h parallel heads, concat + project |
Different representation subspaces |
| Positional Encoding | Sinusoidal fixed frequencies | No learned params, extrapolates |
| Position-wise FFN | 2 linear layers + ReLU, d_{ff}=2048 |
Applied per position independently |
| Residual + LayerNorm | Post-sublayer: LN(x + Sublayer(x)) |
Stabilizes deep stacks |
Technical Deep Dive
Formulation
Scaled Dot-Product Attention:
Attention(Q, K, V) = softmax((QK^T)/(sqrt(d_k)))V
Multi-Head Attention:
MultiHead(Q,K,V) = Concat(head_1,...,head_h)W^O
head_i = Attention(QW^Q_i, KW^K_i, VW^V_i)
FFN:
FFN(x) = max(0, xW_1 + b_1)W_2 + b_2
Encoder Layer:
x' = LN(x + MHSA(x))
x'' = LN(x' + FFN(x'))
Decoder Layer: Adds cross-attention between masked self-attn and FFN.
Training Details
| Hyperparameter | Base | Big |
|---|---|---|
Layers (N) |
6 | 6 |
d_{model} |
512 | 1024 |
d_{ff} |
2048 | 4096 |
Heads (h) |
8 | 16 |
d_k = d_v |
64 | 64 |
| Dropout | 0.1 | 0.3 |
| Label Smoothing | 0.1 | 0.1 |
| Batch Tokens | ~25K | ~25K |
| Steps | 100K | 300K |
| GPUs | 8×P100 | 8×P100 |
| Time | 12 hrs | 3.5 days |
Optimizer: Adam (beta_1=0.9, beta_2=0.98, epsilon=10^{-9})
LR Schedule: lrate = d_{model}^{-0.5} · min(step^{-0.5}, step · warmup^{-1.5}), warmup=4000
Results Summary
Main Results (Table 2)
| Model | EN-DE BLEU | EN-FR BLEU | Training Cost (FLOPs) |
|---|---|---|---|
| ByteNet | 23.75 | — | — |
| Deep-Att + PosUnk | — | 39.2 | 1.0 × 10^{20} |
| GNMT + RL | 24.6 | 39.92 | 2.3 × 10^{19} |
| ConvS2S | 25.16 | 40.46 | 9.6 × 10^{18} |
| MoE | 26.03 | 40.56 | 2.0 × 10^{19} |
| Transformer (base) | 27.3 | 38.1 | 3.3 × 10^{18} |
| Transformer (big) | 28.4 | 41.8 | 2.3 × 10^{19} |
Key takeaway: Transformer (big) beats all prior ensembles on EN-DE (+2.1 BLEU), matches best single on EN-FR at 1/4 training cost of GNMT.
Ablation Studies (Table 3)
| Variation | PPL (dev) | BLEU (dev) | Insight |
|---|---|---|---|
| Base (h=8, d_k=64) | 4.92 | 25.8 | Reference |
| h=1 (single head) | 5.29 | 24.9 | -0.9 BLEU — multi-head critical |
| h=4, d_k=128 | 5.00 | 25.5 | |
| h=16, d_k=32 | 4.91 | 25.8 | |
| h=32, d_k=16 | 5.01 | 25.4 | Too many heads hurt |
| d_k=16 (B) | 5.16 | 25.1 | Small d_k hurts — compatibility needs capacity |
| Learned PE (E) | 4.92 | 25.7 | ≈ Sinusoidal |
| No dropout | 5.77 | 24.6 | Dropout essential |
Parsing Transfer (Table 4)
- 4-layer Transformer,
d_{model}=1024 - WSJ only: 91.3 F1 (vs 90.4 prior best discriminative)
- Semi-supervised: 92.7 F1 (SOTA at time)
Strengths
- Parallelization — Full sequence processed simultaneously
- Long-range deps — Constant path length between any two tokens
- Interpretability — Attention heads learn linguistic structure (syntax, coreference)
- Training efficiency — 12h on 8 P100s for base model
- Generalization — Works on parsing with minimal changes
Weaknesses / Limitations (Honest Assessment)
- Quadratic complexity —
O(n^2 d)memory/compute limits context length - No inductive bias — Needs massive data; no built-in locality/translation equivariance
- Position encoding — Sinusoidal fixed; learned PE doesn't extrapolate well
- Encoder-decoder asymmetry — Decoder can't attend to future (by design), but encoder sees all
- beam search inference — Still autoregressive in decoder, sequential generation
Reproducibility / Implementation Notes
- Compute: Base model feasible on single 24GB GPU with gradient accumulation
- Key hyperparams: Label smoothing 0.1, dropout 0.1, LR warmup critical
- Gotchas:
- Initialize bias in output projection to 0 (not standard)
- Share embedding + output projection weights
- Scale embeddings by
sqrt(d_{model)}
- Open source: tensor2tensor (TF1), fairseq, OpenNMT, HuggingFace
transformers
Connections to Your Wiki
Concepts This Paper Introduces / Advances
- transformer-architecture — Full architecture specification
- attention-mechanism — Scaled dot-product attention formulation
- multi-head-attention — Parallel heads with subspace projections
- positional-encoding — Sinusoidal fixed positional encodings
Concepts This Paper Builds On
- Bahdanau attention (encoder-decoder attention)
- Residual connections (He et al., 2016)
- Layer normalization (Ba et al., 2016)
- Adam optimizer (Kingma & Ba, 2015)
Potential Project Ideas
- efficient-vision-transformer — Apply Transformer to vision (ViT came 2020)
- vision-transformer-dsa-integration — Replace dense attention with DSA
- FlashAttention integration — IO-aware kernel for long-context Transformer
Sources
- arxiv-1706.03762: Primary source — all claims trace to specific sections