Positional Encoding

Summary: Since the Transformer has no recurrence or convolution, it lacks inherent understanding of token order. Positional encodings inject sequence position information into token embeddings. The original paper uses fixed sinusoidal functions, while learned embeddings work equally well.

Problem

In plain English: Self-attention is permutation-equivariant — shuffling input tokens shuffles output identically. Without position info, "dog bites man" and "man bites dog" are indistinguishable.

Sinusoidal Positional Encoding (Original Transformer)

Formulation

For position pos and dimension i:

PE_{(pos, 2i)} = sin((pos)/(10000^{2i/d_{model)}})

PE_{(pos, 2i+1)} = cos((pos)/(10000^{2i/d_{model)}})

Properties

1. Deterministic — No learnable parameters
2. Extrapolation — Can generalize to sequence lengths > training max
3. Linear relationships — For fixed offset k, PE_{pos+k} is linear function of PE_{pos}
4. Unique per position — Different positions have different encoding vectors

Wavelengths

• Dimension i corresponds to wavelength lambda_i = 2pi · 10000^{2i/d_{model}}
• Range: 2pi to 10000 · 2pi ≈ 62831
• Covers both very local and very global positional patterns

Learned Positional Embeddings

Alternative: PE ∈ R^{L_{max} × d_{model}} as learnable parameters.

Table 3 Row (E) result: Learned embeddings achieve 25.7 BLEU vs sinusoidal 25.8 BLEU — essentially identical.

Advantages:

• Can learn task-specific positional patterns
• Simpler implementation

Disadvantages:

• Fixed max sequence length L_{max}
• Cannot extrapolate beyond training length
• More parameters (L_{max} × d_{model})

Variants & Extensions

Method	Key Idea	Extrapolation
Sinusoidal (original)	Fixed sine/cosine	✓ Yes
Learned	Learnable embedding table	✗ No
RoPE (Su et al., 2021)	Rotary position embedding via complex rotation	✓ Good
ALiBi (Press et al., 2021)	Linear bias on attention scores	✓ Yes
Relative PE (Shaw et al., 2018)	Encode relative distances	Varies
T5 Relative Bias	Learned relative position buckets	Limited

RoPE (Rotary Position Embedding)

• Applies rotation to Q,K in complex space
• RoPE(x, pos) = x · e^{i · pos · theta}
• Used in: LLaMA, GPT-NeoX, PaLM, GLM
• Key benefit: Relative position naturally encoded in attention

ALiBi (Attention with Linear Biases)

• Adds static bias -(j-i) to attention scores before softmax
• No positional embeddings added to input
• Enables training on short sequences, inference on longer
• Used in: BLOOM, MPT

Implementation (Sinusoidal)

def get_sinusoidal_encoding(max_len, d_model):
    pe = torch.zeros(max_len, d_model)
    position = torch.arange(0, max_len).unsqueeze(1)
    div_term = torch.exp(torch.arange(0, d_model, 2) * -(math.log(10000.0) / d_model))
    pe[:, 0::2] = torch.sin(position * div_term)
    pe[:, 1::2] = torch.cos(position * div_term)
    return pe.unsqueeze(0)  # (1, max_len, d_model)

Addition to Embeddings

In Transformer:

x_{input} = TokenEmbedding(x) + PositionalEncoding(x)

Scale factor: sqrt(d_{model)} applied to token embeddings before adding PE.

Related Concepts

• transformer-architecture — Where PE is used
• attention-mechanism — Why position info is needed

Sources

• arxiv-1706.03762: Section 3.5 — Sinusoidal PE formulation; Table 3(E) — Learned vs sinusoidal comparison