Noam Shazeer

Summary: Co-author of "Attention Is All You Need"; proposed scaled dot-product attention and multi-head attention. Pioneer of Mixture-of-Experts (MoE) for LLMs. Co-founder of Character.AI (acquired by Google 2024).

Details

For People

  • Affiliation: Character.AI (Co-founder, 2021–2024); Google (2024–present, re-acquired); formerly Google Brain (2012–2021)
  • Key Contributions:
    • Scaled dot-product attention (Transformer paper)
    • Multi-head attention (Transformer paper)
    • Mixture-of-Experts (MoE) — "Outrageously Large Neural Networks" (2017)
    • GLU variants, T5 architecture, PaLM contributions
    • Character.AI — Conversational LLM platform
  • Website: [Character.AI] / [Google Scholar]
  • Notable Works:
    • "Attention Is All You Need" (NeurIPS 2017)
    • "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer" (ICLR 2017)
    • "GLU Variants Improve Transformer" (2020)
    • T5: "Exploring the Limits of Transfer Learning" (2019)

Significance

Co-inventor of core Transformer components. MoE is now standard in frontier LLMs (GPT-4, PaLM-2, DeepSeek, Mixtral).

For your research:

  • Sparse attention / MoE connection: Your DSA-ViT explores sparse attention — Shazeer pioneered sparse computation (MoE)
  • Efficiency lineage: FlashAttention, DSA, MoE all address same bottleneck Shazeer identified early
  • PhD narrative: "From dense attention (Vaswani/Shazeer) → sparse computation (MoE) → dynamic sparse attention (DSA)"

Related Concepts

Sources

  • arxiv-1706.03762: Author contributions note his proposal of scaled dot-product & multi-head attention