Author(s): Naimish Baranwal

Originally published on Towards AI.

It has been several years since the GPT-2 era — from 2019 to 2026 — and yet, one of the most interesting things about the rapid evolution of AI is that the core idea behind the Transformer architecture has remained remarkably consistent.

In an industry where technologies can become obsolete within a few years, it is almost fascinating that the fundamental Q (Query), K (Key), and V (Value) attention mechanism used by the Transformer is still at the heart of today’s most powerful Large Language Models (LLMs).

You could almost call it a stroke of genius that researchers came up with such a simple yet powerful mechanism — and that this idea has continued to scale with us for years.

However, saying that the Transformer architecture has remained the same does not mean that there has been little innovation. Quite the opposite. A tremendous amount of research has gone into improving the Transformer around its core architecture.

Instead of completely replacing the fundamental Q, K, V attention mechanism, researchers have focused on solving the limitations that appear when Transformers are scaled to billions or trillions of parameters and deployed in real-world applications.

Some of the major areas of improvement include:

• Inference efficiency — such as KV Cache, which significantly reduces redundant computation during autoregressive generation.
• Attention optimization — including MHA, MQA, GQA, MLA, Flash Attention, and other approaches designed to make attention more memory- and compute-efficient.
• Model capacity and parameter efficiency — such as Mixture of Experts (MoE).
• Training stability and architecture improvements — including Pre-LN, RMSNorm, QK-Norm, and SwiGLU.
• Positional encoding and long-context handling— Sinusoidal, RoPE, YaRN (Yet another RoPE extensioN), and LongRoPE.
• Alternative sequence modeling approaches — such as State Space Models (SSMs) and Mamba, which explore whether attention is always necessary for efficient sequence modeling.

In this series of articles, we are going to walk through each of these major milestones one by one. Whether you are a beginner or a seasoned engineer, this article is for you. For every upgrade, we will start with a simple, non-technical analogy to build intuition, and then we will pop the hood and dive into the advanced mathematical formulas that make it all work.

First, let us start with positional embedding development:

Positional Embeddings: Sinusoidal & RoPE (Rotary Position Embedding)

The Intuition: Why Position Matters. In language, word order is everything. Take the sentences “The bat killed the man” and “The man killed the bat”. The words are exactly the same, but the meaning is completely reversed based purely on their positions. Because the core Q, K, V mechanism of a Transformer processes all words simultaneously (unlike older sequential RNNs), it has no built-in concept of order. If we don’t explicitly tell the model where the words are, it just sees a jumbled “bag of words.”

The idea was straightforward: take the token’s representation and add a separate positional vector to it.

For example, suppose the embedding of a word is:

[1.2, 0.5, 0.1]
and the positional encoding for position 3 is:[0.1, 0.1, 0.1]
The model simply adds them together:
[1.2, 0.5, 0.1]
+
[0.1, 0.1, 0.1]
----------------
[1.3, 0.6, 0.2]

The resulting vector therefore contains both semantic information about the token and information about its position in the sequence.

This is what we mean by an absolute positional encoding: the representation is explicitly informed that the token is at position 3, position 10, position 100, and so on.

The Original Approach: Sinusoidal Positional Encoding

In the original Transformer paper (Attention Is All You Need), the authors introduced sinusoidal positional encodings to provide the model with information about the position of each token.

The equations used in the original paper were:

PE(pos, 2i) = sin(pos / 10000^(2i / d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i / d_model))
pos = position of the token
i = dimension index
d_model = embedding dimension
For a given dimension:
angle = pos(p) × frequency(w)
where the frequency is roughly:
frequency(w) ∝ 1 / 10000^(2i/d)

Here, i is the embedding dimension index; for a 768-dimensional embedding vector (the same as the word embedding dimension), there are 768 indices, and p is the token's position.

In any embedding, we need,

• Higher-index dimensions → lower frequency → change more slowly with position
• Lower-index dimensions → higher frequency → change more rapidly with position

Lower-index dimensions change rapidly with position and therefore provide fine-grained positional information, while higher-index dimensions change more slowly and provide coarse-grained positional information. Together, these different frequencies give the model a multi-scale representation of position.

Consider:

“The company opened a new office in Delhi, and the company opened another office in Mumbai.”

Suppose the model encounters the word “office” twice.

office₁ → position 6
office₂ → position 13

We want the positional representation to capture two different things:

Fine-grained position: The nearby positions should be distinguishable:

position 5
position 6
position 7

A rapidly changing sinusoidal dimension is useful here because even a small change in position produces a noticeable change:

5 → 6 → 7 → 8

So the model gets a fine-grained positional signal.

Coarse-grained position

Now consider:

position 5
position 25
position 45
position 65

We don’t necessarily need every position to look completely different in every dimension. Some slowly varying dimensions can provide a coarse sense of where we are in the sequence.

So you can think of sinusoidal encoding as giving the model a collection of rulers:

Fast frequency
→ centimeter ruler
→ distinguishes nearby positions
Medium frequency
→ meter ruler
→ captures medium-range position
Slow frequency
→ kilometer ruler
→ captures broad position

Why Look Beyond Additive Positional Encoding?

Adding values changes the magnitude and direction of the original word embedding. One way to think about this is like writing a page number directly over the text of a book with a dark marker.

RoPE: The Core Idea

Introduced in the RoFormer paper (2021) and later made the industry standard by models like LLaMA, PaLM, Mistral, and DeepSeek-V2, RoPE takes a completely different approach.

Instead of adding a positional vector to the token representation, RoPE applies a position-dependent rotation to the Query and Key vectors inside the attention mechanism.

This allows the attention mechanism to take positional relationships into account while calculating how strongly one token should attend to another.

The Deep Dive: How RoPE Works Mathematically

Let’s build this idea mathematically.

1. The Rotation Matrix

Let’s start with the simplest possible case. Suppose we have a 2-dimensional vector:

x = [x₁, x₂]
A standard 2D rotation by an angle θ can be represented using the rotation matrix:
 [ cos(θ) -sin(θ) ]
R(θ) = [ sin(θ) cos(θ) ]
Applying this rotation matrix to our vector gives:x' = R(θ)x
Therefore:

 [ cos(θ) -sin(θ) ] [ x₁ ]
x' = [ sin(θ) cos(θ) ] × [ x₂ ]
When we perform the matrix multiplication:
x'₁ = x₁cos(θ) - x₂sin(θ)
x'₂ = x₁sin(θ) + x₂cos(θ)
So the rotated vector becomes:
x' = [ x₁cos(θ) - x₂sin(θ)]
 [x₁sin(θ) + x₂cos(θ)]
The original magnitude is: ||x|| = √(x₁² + x₂²)
After rotation: ||x'|| = √[(x₁cos(θ) - x₂sin(θ))²+(x₁sin(θ) + x₂cos(θ))²]
=√(x₁² + x₂²)

||x'|| = ||x||; Here magnitude remains same

If the token occurs at position m,
RoPE rotates the vector according to that position.
We can write the rotation angle as: mθ
So the position-dependent rotation matrix becomes:

 [ cos(mθ) -sin(mθ) ]
Rₘ = [ sin(mθ) cos(mθ) ]

x' =[x₁cos(mθ) - x₂sin(mθ)]
 [x₁sin(mθ) + x₂cos(mθ)]

The key point is that rotation changes the vector's direction but not its magnitude, and it rotates according to its position.

For example:

Position 1 → rotate by θ
Position 2 → rotate by 2θ
Position 3 → rotate by 3θ
Position 4 → rotate by 4θ

So position is encoded through the orientation of the vector. But Real LLMs Have Thousands of Dimensions. A modern Transformer does not have a 2-dimensional Query or Key vector.

Suppose a model has dimensions:
d=4096
Then a Query vector could look conceptually like:
Q = [q₁, q₂, q₃, q₄, q₅, q₆, ..., q₄₀₉₆]
RoPE divides these dimensions into pairs:
(q₁, q₂)
(q₃, q₄)
(q₅, q₆)
...

Each pair behaves like a small 2D vector.
(q₁, q₂) → rotation using θ₁
(q₃, q₄) → rotation using θ₂
(q₅, q₆) → rotation using θ₃
...

The important detail is that different dimension pairs use different frequencies. So we do not actually have one single θ. Instead, each pair gets its own angular frequency.

So conceptually:

Here, Rm represents the RoPE rotation operator corresponding to token position m, while Rn represents the rotation operator corresponding to position n. In a real Transformer, these operators consist of multiple 2D rotations applied to pairs of dimensions, with each pair using a different frequency.

For position m, the angle for the i-th pair can be written as:

θₘ,ᵢ = m × ωᵢ
where
m = token position
i = dimension-pair index
ωᵢ = frequency associated with that dimension pair
d = embedding dimension

A common formulation is:
ωᵢ = 1 / base^(2i / d)
θₘ,ᵢ = m / base^(2i / d)

This means that some dimensions rotate quickly while others rotate slowly. 
Conceptually:
Low-index dimensions
 ↓
Higher frequency
 ↓
Rapid change with position
 ↓
Fine-grained positional information
------------------------------------------------
High-index dimensions
 ↓
Lower frequency
 ↓
Slow change with position
 ↓
Coarse-grained positional information

So RoPE provides a multi-scale representation of position.

2. Where Exactly Is RoPE Applied?

This is an important distinction.RoPE is not normally applied directly to the original token embedding. Instead, the Transformer first produces Query and Key vectors:

So if:

Qₘ = Query at position m
Kₙ = Key at position n

RoPE transforms them into:

Q'ₘ = Rₘ Qₘ
K'ₙ = Rₙ Kₙ

where Rₘ and Rₙ are the position-dependent rotation operators.
 [ cos(mθ) -sin(mθ) ]
Rₘ = [ sin(mθ) cos(mθ) ] 

 [ cos(nθ) -sin(nθ) ]
Rₙ = [ sin(nθ) cos(nθ) ]
Now comes the interesting part.
The Magic Happens in the Dot Product
Let's calculate the attention score between:

Query at position m and Key at position n
After applying RoPE, the attention score becomes:
Score(m,n) = (Q'ₘ)ᵀK'ₙ
Substituting:
Q'ₘ = RₘQₘ
K'ₙ = RₙKₙ
Score(m,n)=(RₘQₘ)ᵀ(RₙKₙ)
Now use the transpose property:
(AB)ᵀ = BᵀAᵀ
Therefore:
Score(m,n)=QₘᵀRₘᵀRₙKₙ

Now we use an important property of rotation matrices. 
The transpose of a rotation is equivalent to rotating in the opposite direction:
Rₘᵀ = R₋ₘ
RₘᵀRₙ=R₋ₘRₙ
R₋ₘRₙ = Rₙ₋ₘ
Therefore:
Score(m,n)=QₘᵀRₙ₋ₘKₙ

This is the key equation. Notice what appeared:Rₙ₋ₘ

which is the relative position between the two tokens.

3. Why Is This Important?

Suppose we have:

Token A → position 10
Token B → position 13
Their relative distance is: 13 - 10 = 3
Now suppose the same two tokens appear at:
Token A → position 100
Token B → position 103

Their relative distance is still:103 - 100 = 3

With RoPE, the positional transformation involved in their Q-K interaction is associated with the same relative offset: R₃ in both cases. That is the mathematical intuition behind RoPE’s relative positional behavior.

The absolute positions are still used to construct the rotated Q and K vectors, but when those vectors interact through the dot product, the relative offset naturally appears in the mathematical expression.

4. The Mathematical Example: How the Whole Combination Works

Let’s make this much more concrete.

Let’s look at two concrete numerical examples to understand RoPE.

The first example shows how RoPE rotates a complete query vector.
The second shows how this rotation affects the Q-K dot product and creates relative positional information.

──────────────────────────────────────────────
1. COMPLETE 4D Query Rotation
──────────────────────────────────────────────

Suppose we have a 4-dimensional Query vector for a token at position m = 2:

q = [1.0, 0.0, 1.0, 0.0]

Since we have 4 dimensions, RoPE splits the vector into two 2D pairs:

Pair 1 → [1.0, 0.0]
Pair 2 → [1.0, 0.0]

For simplicity, assume:

Pair 1 → rotates 90° per position
Pair 2 → rotates 45° per position

At position m = 2:

Pair 1 angle = 2 × 90° = 180°
Pair 2 angle = 2 × 45° = 90°


Pair 1:

Original = [1.0, 0.0]

Rotation matrix for 180°:

[ -1 0 ]
[ 0 -1 ]

Applying the rotation:

[ -1 0 ] [ 1.0 ] [ -1.0 ]
[ 0 -1 ] [ 0.0 ] = [ 0.0 ]

Result → [-1.0, 0.0]


Pair 2:

Original = [1.0, 0.0]

Rotation matrix for 90°:

[ 0 -1 ]
[ 1 0 ]

Applying the rotation:

[ 0 -1 ] [ 1.0 ] [ 0.0 ]
[ 1 0 ] [ 0.0 ] = [ 1.0 ]

Result → [0.0, 1.0]


Recombine the two pairs:

q_rotated = [-1.0, 0.0, 0.0, 1.0]


Did the rotation change the magnitude?

Original:

||q|| = √(1² + 0² + 1² + 0²)
 = √2

Rotated:

||q_rotated|| = √((-1)² + 0² + 0² + 1²)
 = √2

Therefore:

||q_rotated|| = ||q||

The direction changed according to the token’s position, but the magnitude remained unchanged.


──────────────────────────────────────────────
2. HOW DOES THIS CREATE RELATIVE POSITION?
──────────────────────────────────────────────

Now let’s use a simpler 2D example to understand the most important property of RoPE.

Suppose:

Q = [1, 0]
K = [1, 0]

Without positional rotation:

Q · K = (1 × 1) + (0 × 0)
 = 1

The two vectors are perfectly aligned.


Now suppose:

Query position m = 0
Key position n = 90

For simplicity, assume a rotation frequency of 1° per position.

Therefore:

Query rotation = 0 × 1° = 0°
Key rotation = 90 × 1° = 90°

So:

Q' = [1, 0]

K' = [0, 1]

Now calculate their dot product:

Q' · K'
= (1 × 0) + (0 × 1)
= 0

The vectors are now perpendicular.

Why did the dot product change?

Because the Query and Key have a relative positional difference:

n - m = 90 - 0
 = 90°


Now move both tokens forward by the same amount:

Query position = 20
Key position = 110

Their absolute positions have changed, but their relative distance has not:

110 - 20 = 90

Therefore, both vectors rotate further, but the angle between them remains the same.

This is the key intuition behind RoPE:

The absolute position determines how much each vector rotates,
while the difference between the two positions determines their
relative rotation.

In mathematical form:

Q'ₘ = Rₘ Qₘ
K'ₙ = Rₙ Kₙ

and their interaction becomes:

(Q'ₘ)ᵀK'ₙ = Qₘᵀ Rₘᵀ Rₙ Kₙ

Since:

RₘᵀRₙ = Rₙ₋ₘ

we get:

(Q'ₘ)ᵀK'ₙ = Qₘᵀ Rₙ₋ₘ Kₙ

The important term is:

Rₙ₋ₘ

which depends on the relative position (n - m).

This is the mathematical reason RoPE can inject relative positional information directly into the Query-Key interaction.

Both vectors rotate, but the relative rotation between them remains the same. That is the core geometric intuition behind RoPE.

We can now summarize the entire mechanism.

**Traditional additive positional encoding**
The idea is:
Token Representation + Position Information
 ↓
 Modified representation

**RoPE**
The idea is:
Query + Position → Rotate Query
Key + Position → Rotate Key
 ↓
 Rotated Q · Rotated K
 ↓
 Position-aware attention

Mathematically:
Q'ₘ = RₘQₘ,K'ₙ = RₙKₙ
and therefore:
(Q'ₘ)ᵀK'ₙ=QₘᵀRₙ₋ₘKₙ
The appearance of:n - m is the key mathematical insight.

TL;DR: How RoPE Works

We rotate each Query and Key vector based on its position in the sequence. This changes the internal values of the vector to track its position, but perfectly preserves the overall magnitude of the embedding. By applying fast rotations to some dimensions and slow rotations to others, the model captures both fine-grained (short) and coarse-grained (long) positional relationships.

When the rotated Q and K vectors interact through the attention dot product, the rotation matrices combine such that the positional component of their interaction depends on the relative position between the two tokens rather than their absolute positions. This allows RoPE to inject relative positional information directly into the attention mechanism without simply adding a positional vector to the token representation.

Imagine a model trained with a context length of 1,024 tokens. What happens if we want that same model to work with 128K tokens? Increasing the maximum context length alone does not guarantee that the model will understand positional information correctly at these new positions. For RoPE-based models, the positional frequencies need to be carefully rescaled or extended so that the model can generalize beyond its original training range. This is where techniques such as YaRN(Yet another RoPE extensioN) and LongRoPE become important.

Extending the Context: The Battle for Infinite Memory

Imagine a model trained with a context length of 1,024 tokens. What happens if we want that same model to work with 128K tokens? Increasing the maximum context length alone does not guarantee that the model will understand positional information correctly at these new positions.

For RoPE-based models, when the position offset exceeds the training context length, the model’s performance decreases drastically due to “unseen” rotation patterns. The positional frequencies need to be carefully rescaled or extended so that the model can generalize beyond its original training range.

This is where techniques such as YaRN (Yet another RoPE extensioN) and LongRoPE become important.

Position Interpolation (PI)

One simple and effective idea is to rescale the position into the training context length. This is called Position Interpolation. For example, if we want to extend the context from 4K to 20K, we scale the position by 1/5. This effectively slows down all frequencies by a factor of 5. However, as we discussed earlier, high-frequency components are critical for distinguishing nearby tokens.

s = L_new / L_train
p_scaled = p / s​

L_train = original training context length
L_new = desired new context length
p = original token position
s = context extension scale factor
p_scaled = rescaled position

L_train = 8K
L_new = 32K

s = 32K / 8K
 = 4

p_scaled = p / 4

YaRN (NTK-Aware Scaling)

YaRN applies non-uniform scaling across RoPE frequencies. The goal is to preserve useful short-range positional resolution while compressing the frequencies that would otherwise produce problematic phase behavior at longer positions.

• High Frequencies: The scaling factor is close to 1, meaning we keep the high-frequency components exactly as they are to preserve fine-grained, local attention.
• Low Frequencies: We apply a heavy scaling factor (like 1/5) to compress the massive new document length into the model’s known mathematical space.

LongRoPE: Pushing to 2 Million Tokens

LongRoPE takes this concept even further by abandoning uniform mathematical scaling entirely. Instead of using a fixed equation, LongRoPE uses an evolutionary search algorithm to find the absolute effective scaling factor for every single dimension individually. Its proposed progressive extension strategy was evaluated for context windows beyond 2 million tokens, while also attempting to recover short-context performance..

Conclusion to Part 1

We have now solved the mapping problem. RoPE and its extensions give the model a mechanism for representing token positions across much longer contexts. However, a larger context window does not automatically guarantee perfect retrieval or understanding of every token.

But this massive context window creates a terrifying new bottleneck: Memory and compute.

When a model generates text autoregressively (one word at a time), it has to recalculate the attention scores for every previous word in the sequence, over and over again. If you have 100,000 words in your context, doing this mathematical recalculation for every new token is computationally impossible for real-time chat.

To solve this, researchers had to invent a way to cache the model’s memory, leading to the most important inference optimization in modern AI, and the structural changes required to make it fit on a GPU.

In Part 2, we will dive into the KV Cache and how it forced the evolution of the Transformer’s attention mechanism from Multi-Head Attention (MHA) into Grouped Query Attention (GQA) and Multi-Head Latent Attention (MLA).

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI