BERT

Summary: BERT (Bidirectional Encoder Representations from Transformers) — encoder-only Transformer pretrained with masked language modeling (MLM) and next sentence prediction (NSP). Foundation for modern NLP; spawned RoBERTa, DeBERTa, ELECTRA, etc.

Details

For Models

  • Architecture: Transformer Encoder (12L/24L, 768/1024 hidden, 12/16 heads)
  • Parameters: Base 110M, Large 340M
  • Pretraining:
    • MLM: Mask 15% tokens → predict masked tokens (bidirectional context)
    • NSP: Predict if sentence B follows sentence A (removed in later variants)
  • Data: BooksCorpus (800M words) + English Wikipedia (2.5B words)
  • License: Apache 2.0 (Google Research)

Variants Timeline

Model Year Key Change
BERT 2018 Original
RoBERTa 2019 Remove NSP, dynamic masking, more data, larger batches
ALBERT 2019 Parameter sharing, factorized embedding, SOP loss
ELECTRA 2020 Replaced token detection (GAN-style) — more efficient
DeBERTa 2020 Disentangled attention, enhanced mask decoder
BERT-wwm 2019 Whole word masking (Chinese)

Significance

Encoder-only paradigm: Proved bidirectional pretraining > autoregressive (GPT) for NLU tasks (GLUE, SQuAD).

For your research:

  • ViT connection: ViT = BERT for images (patches as tokens, [CLS] pooling)
  • Efficient fine-tuning: LoRA/QLoRA originally validated on BERT-style models
  • Distillation: BERT → smaller students (TinyBERT, DistilBERT) — relevant for edge deployment

Related

Sources

  • public-knowledge: Devlin et al., "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (NAACL 2019)