NoPE (No Positional Embeddings) Attention

Summary: NoPE refers to using content-only attention (no positional encoding) in selected layers — typically global attention layers where semantic similarity matters more than position. In Inkling, global layers apply learned relative bias only to the preceding 1024 tokens; beyond that, attention becomes effectively NoPE (content-only).

Overview

The NoPE conjecture: Positional encoding is critical for local/syntactic processing but detrimental or unnecessary for global/semantic processing. By removing position bias from global attention, models can better match semantically similar but positionally distant tokens (e.g., repeated motifs, cross-document references).

Key Components

NoPE in Global Layers (Inkling Style)

  • Local layers (55): Full relative position bias within 512-token window
  • Global layers (11): Relative position bias for first 1024 preceding tokens only
  • Beyond 1024: No positional bias → pure content-based attention (Attn(Q,K,V) = softmax(QK^T/√d)V)

Why NoPE Works for Global Attention

  1. Semantic matching: Global attention often retrieves by meaning, not position
  2. Extrapolation: No positional params to extrapolate
  3. Efficiency: Simpler kernel (no bias add for long range)
  4. Regularization: Forces content-based retrieval for long-range

Mathematical Formulation

In plain English: Standard attention adds position bias to QK^T scores. NoPE skips this addition for long-range pairs — it just uses raw similarity of content representations.

# Standard attention with bias
scores = Q @ K.transpose(-2,-1) / sqrt(d) + bias  # bias = f(i,j)

# NoPE for |i-j| > threshold (global layer)
if distance > 1024:
    scores = Q @ K.transpose(-2,-1) / sqrt(d)  # no bias
else:
    scores = Q @ K.transpose(-2,-1) / sqrt(d) + bias

Variants / Extensions

  • Full NoPE: No positional encoding anywhere (rare, usually fails on local tasks)
  • Hybrid NoPE: Local layers with position, global layers without (Inkling, some Longformer variants)
  • Conditional NoPE: Learn to predict when position matters
  • XPos/RoPE with NoPE tail: RoPE for local, disable for far positions

Comparison

Approach Local Layers Global Layers Extrapolation
Standard RoPE RoPE RoPE Poor
ALiBi ALiBi ALiBi Good
NoPE (full) None None Good (but local suffers)
Inkling Hybrid Rel Bias (512) Rel Bias (1024) → NoPE Claimed best

Applications

  • Inkling (11/66 global layers use NoPE beyond 1024)
  • Long-context models (>100K tokens)
  • Retrieval-augmented models (global = retrieval)
  • Models with sliding window + global token patterns

Historical Context

Year Work Contribution
2017 Transformer Absolute sinusoidal everywhere
2019 T5 Learned relative bias everywhere
2021 RoPE Rotary everywhere
2021 ALiBi Linear bias everywhere
2022 Longformer Local + global tokens (global has pos)
2022 BigBird Sparse + random + global
2023 NoPE (various) Remove pos from global / certain heads
2026 Inkling Hybrid: 55 local (rel bias), 11 global (bias→NoPE)

Related Concepts

Sources

  • inkling-raschka-blog: Architecture observations #3 — Global layers: learned bias only for preceding 1024 tokens; beyond that effectively content-based (NoPE intuition)