Learned Relative-Position Bias

Summary: An input-dependent, learned relative position bias replacing RoPE in Inkling. Applied within local attention windows (512 tokens) and limited to 1024 tokens in global layers. Claimed to perform better and extrapolate better to longer sequences than RoPE.

Overview

RoPE (Rotary Positional Embedding) has become the default for LLM positional encoding, but it has known limitations: fixed frequency bases, extrapolation degradation beyond training length, and coupling of position with content. Inkling uses a learned relative-position bias — a classical idea revived with modern scale and input-dependent conditioning.

Key Components

Relative Position Bias (Classical)

  • Attention bias: B[i,j] = w_{i-j} where w is learned
  • Depends only on relative distance i-j
  • Used in T5, DeBERTa, early Transformer variants

Inkling's Learned Input-Dependent Version

  • Input-dependent: Bias can be conditioned on input content (not fully specified in public details)
  • Local layers: Applied within 512-token sliding window
  • Global layers: Applied to preceding 1024 tokens only; beyond that → content-only attention (NoPE-like)
  • Claim: Better performance + better extrapolation than RoPE

Hybrid Attention Interaction

  • 55 local layers: Standard relative bias in 512 window
  • 11 global layers: Relative bias for 1024 tokens → then NoPE (content-only)
  • Rationale: Global layers attend to far positions where precise position matters less than content similarity

Mathematical Formulation

In plain English: Standard RoPE rotates Q/K vectors by position-dependent angles. Learned relative bias instead adds a scalar bias b[i-j] to attention scores for each relative distance. Inkling makes this learned and potentially input-dependent, and only applies it locally.

Standard Relative Bias (T5-style)

Attn(Q,K,V)_{i,j} = (Q_i K_j^T)/√d + b_{i-j}
b_d = learned_embedding[d + max_distance]  # d = i-j

Inkling's Input-Dependent Version (Conceptual)

# Local layers: distance-based, potentially input-conditioned
b_{i,j} = f_θ(x_i, x_j, i-j)  for |i-j| < 512

# Global layers: limited range
b_{i,j} = f_θ(x_i, x_j, i-j)  for |i-j| < 1024
b_{i,j} = 0                      for |i-j| ≥ 1024  # NoPE-style

Where f_θ is a learned function (could be MLP on concatenated embeddings + distance encoding).

Comparison: RoPE vs Learned Relative Bias

Property RoPE Learned Relative Bias (Inkling)
Parametrization Fixed frequencies Learned embeddings
Extrapolation Degrades beyond train length Claimed better (local bias)
Content coupling Position baked into Q/K Additive bias, separable
Long-range Global by design Limited window → content-only
Parameters Minimal (freq bases) O(max_dist) or input-dep
Implementation Kernel-fused Standard bias add

Extrapolation Analysis

RoPE extrapolation failure modes:

  1. Unseen frequencies: cos/sin angles outside training range
  2. Position-content entanglement: Can't disentangle novel positions

Learned bias advantages:

  1. Local inductive bias: Nearby tokens should interact strongly regardless of absolute position
  2. Saturation: Bias plateaus at window edge (or cuts off) — no wild extrapolation
  3. Content fallback: Beyond window, pure content attention (NoPE) works if semantic similarity suffices

Variants / Extensions

  • ALiBi: Fixed linear bias b_{i-j} = -m * |i-j| (no learning)
  • T5 Bias: Learned relative buckets (logarithmic)
  • DeBERTa: Disentangled attention with content-to-position + position-to-content
  • NoPE (No Positional Encoding): Content-only attention in selected layers
  • Inkling: Learned + input-dependent + hybrid windowed

Applications

  • Inkling (975B MoE, 1M context)
  • Models targeting extreme context extrapolation
  • Hybrid architectures (local + global attention)
  • Scenarios where RoPE extrapolation is problematic

Historical Context

Year Work Contribution
2017 Transformer Absolute sinusoidal
2018 Shaw et al. Relative position representations
2019 T5 Learned relative bias buckets
2020 DeBERTa Disentangled attention
2021 RoPE (Su et al.) Rotary, multiplicative, extrapolation
2021 ALiBi (Press et al.) Linear bias, strong extrapolation
2022 NoPE / XPos NoPE in global layers; XPos for RoPE extrapolation
2026 Inkling Learned input-dependent relative bias + hybrid windowed + NoPE global

Related Concepts

Sources

  • inkling-raschka-blog: Architecture observations #3 — Learned input-dependent relative-position bias instead of RoPE; claims better performance and extrapolation; local layers: 512 window; global layers: 1024 bias window then content-only