Understanding Attention From Scratch

I think if you have ever spent even a single day learning about current models, then attention is most probably the first term you may have heard. And when it comes to the architecture of these models, I am 200% sure attention is one of the first things most of us come across.

The most famous paper from 2017 itself says:

"Attention Is All You Need."

And nowadays, we have so many things around attention that you almost can't count them. Self-attention, multi-head attention, FlashAttention, grouped-query attention, sparse attention, paged attention, this attention, that attention :)

So I would say there maybe 0.01% chance that you have NOT heard the word attention in the LLM/Transformer context. There maybe a 33% chance that you know its definition, but you don't know what it is.

And there is a pretty good chance that you know what attention is, its definition, equations, types, but still don't have the complete correct picture of what is actually happening. I am just guessing :). I have not done any research on this data (nor we need).

So in this article, I want to take you get the complete & correct feel of attention from the absolute base via first principles.

We will not start from any definition or equation. We will eventually reach there. I feel, they are of lowest priority. Once you understand the core logic it becomes super-easy to derive them.

First, I want us to understand **why we even need something like attention in the first place. **

What is the actual problem?

What are we trying to achieve or solve?

So, we will start just from normal thinking and intuition, and once that is clear, we will slowly go deeper into what attention is and how it actually works.

And later, we can go further into the some common forms of attention we see in current models, because attention today has become much more sophisticated than the original basic form.

But first, the core needs to be clear. that is imp.

Also, before we start: might be you prefer visual content over reading an article. I actually made a video explaining attention. And this is no promotion at all. If you like videos, you can watch the video. If you prefer text, stay here. simple.

If you ask me, just sit with a calm mind and continue reading. You don't really need to go anywhere unless you genuinely prefer video.

Okay.

Also, this is going to be a slightly longer read, because I have tried to put my best in explaining things from zero level of understanding, so I will go step by step, so u don't feel where this came out the blue. And I hope you expect so :)

---

Let's start with something completely unrelated. Just normal English.

Suppose I give you this sentence:

The animal could not cross the street because it was too tired.

Now one more:

The animal could not cross the street because it was too wide.

Notice one thing. Both sentences contain the word:

it

But in the first sentence, it refers to the animal because:

the animal was too tired.

A street being tired obviously doesn't make sense at all :)

But in the second sentence, it refers to the street because:

the street was too wide.

A wide animal being unable to cross the street doesn't really make sense here.

Now wait for a second and think about this.

How did you know that?

Nobody explicitly told you:

Hey, in sentence one, it means animal.

And:

In sentence two, it means street.

You figured it out yourself. (think how ?)

And these are just two sentences.

There can be millions or billions of sentences containing the token it, and what it actually refers to can be different depending on the sentence.

Let's take another example. Suppose I just give you one word:

bank

What does it mean? Maybe I am talking about the bank where you deposit your money.

Or maybe I am talking about the bank of a river. From the word bank alone, you cannot really know what I mean. Now suppose I give you:

"I deposited some money in the bank."

Okay.

I think now it is pretty obvious what bank means.

Now another one:

"We sat on the bank of the river."

Again, exactly the same word:

bank

But now the meaning is completely different. So again, let's stop here and think.

How did you infer that in the first sentence bank means the financial bank, while in the second sentence it means the side of a river?

Because you did not look at bank alone. You looked at the other words around it.

In the first sentence, something like:

deposited → money → bank

immediately gives you a strong clue about what kind of bank we are talking about.

In the second one:

sat → bank → river

gives you completely different context.

So what did your brain basically do?

It looked at the other words in the sentence, found the information that was relevant to the current word, and used that information to understand what the current word means in this particular context.

And this is the first important idea.

A word or more correctly for an LLM, a token cannot always be understood properly by looking at that token alone.

It needs information from the other tokens around it.

Now let's bring this back to a model. Suppose the model receives:

"I deposited some money in the bank."

At some point inside the model, we have a representation for every token.

Something for I.

Something for deposited.

Something for money.

Something for bank.

and so on.

But if the model wants to build a good representation for bank, it somehow needs to understand:

Okay, which of the other tokens in this sentence are actually useful for understanding me?

money is probably very relevant.

deposited is probably relevant.

Maybe some other token matters less.

And in another sentence:

"We sat on the bank of the river."

the situation changes.

Now river becomes extremely useful for understanding bank.

So what we need is some mechanism through which a token can basically look at the other tokens and figure out:

Which tokens are relevant to me?

And then:

What useful information should I take from them?

That, at the most intuitive level, is what attention is trying to do.

It allows the representation of one token to be influenced by the other relevant tokens in the sequence.

Not every token has to matter equally.

Some may matter a lot.

Some may matter a little.

Some may practically not matter at all.

And importantly, this relationship can change depending on the sentence.

For:

"I deposited money in the bank"

money may have a strong relationship with bank.

For:

"I sat beside the river bank"

river may have a strong relationship with bank.

Same token.

Different context.

Different relationships.

Different final meaning.

Okay.

So conceptually, our problem is becoming pretty clear.

But now we have a harder question.

How does a machine actually do this?

As humans, we can just read the sentence, we know the rules of English grammar. and we can understand that river matters more here.

But the model does not literally see:

• river
• bank
• money

the way we do.

Inside the model (which is just a collection of huge matrices), these tokens are just represented as vectors of numbers.

So somehow, using only these vectors and mathematical operations, the model needs to figure out:

• "What is the meaning of this token in this context here?"
• "How much should this token care about that token?"

And this is where the actual mechanism of attention starts.

(In case you do not understand why are we talking about numbers, matrices, and vectors instead of normal text? Then you should slightly know a bit about model architecture like first we use the tokenizer to convert the text into token ids. then token ids to vector embeddings. then we start processing it transformer's layer by layer)

---

Okay, so now we know the problem.

For every token, we somehow need to figure out which other tokens are relevant to it. (to infer the relevant info of current token)

Let's stay with the same example:

"I deposited some money in the bank."

Suppose right now we are trying to build a better representation for the token bank.

Ideally, we want the model to realize that money and deposited are quite relevant here. Maybe the is useful grammatically, but probably not nearly as useful for understanding what kind of bank we are talking about.

So somehow we need to calculate a relevance between:

bank ↔ I

bank ↔ deposited

bank ↔ some

bank ↔ money

bank ↔ in

bank ↔ the

and so on.

Now remember, inside the model these aren't English words anymore. Every token is represented by a vector of numbers. (we generally say it input matrix X, having vectors of all tokens as its rows)

So our real problem becomes something like:

I have one vector for bank, another vector for money, another for deposited, etc. How can I mathematically measure how relevant one token is to another?

But the next question can be How can I mathematically figure out which of these tokens are relevant to bank?

At first, you may think: why don't we simply compare the vector of bank directly with the vector of money?

And technically, we could design something like that. But think about what we actually want. For the current token bank, we want some representation that basically says:

"What kind of information am I looking for right now?"

And for every other token, we want another representation that basically says:

"What kind of information do I have that another token may care about?"

These are slightly different roles. For example, the original vector of money may contain a lot of information about money. But when we are deciding whether it is relevant to bank, maybe only some parts of that information matter.

Similarly, the vector of bank contains a lot of information too, but for this particular operation we mainly want to know what kind of context bank is looking for.

So instead of comparing the original token vectors directly, we let the model create two different learned views of them.

One view is used for asking:

"What am I looking for?"

We call this the Query Q.

The other view is used for matching:

"What information do I contain that may be relevant?"

We call this the Key K.

So from the same token representation x, the model learns two different transformations:

Now when we want to know whether money is relevant to bank, we compare:

Q_bank with K_money

For deposited:

Q_bank with K_deposited

And so on.

So Query and Key are not some random extra things we suddenly added.

They basically give the model a learned way to separate:

what a token is looking for

from

what another token has to offer for matching.

And once we have these two representations, our next problem becomes much simpler:

How do we compare a Query vector with a Key vector and turn that into a relevance score?

I think we need comparison for something like:

Q_bank with K_I

Q_bank with K_deposited

Q_bank with K_money

Q_bank with K_bank

and so on.

If the Query of bank matches strongly with the Key of money, then maybe money contains information that is useful for bank.

If it matches much less with some other token (like the), then maybe that token is not as important in this particular context.

So Query and Key are basically giving us a way to answer our first question:

"How much should I care about this token?"

Now the next question is obvious:

How do we compare two vectors?

One of the simplest ways is the dot product.

Suppose, just for intuition, that our Query and Key vectors were extremely tiny:

and:

Their dot product would be:

Now suppose another token has:

Then:

Again, don't focus on these exact numbers. Real attention vectors are obviously much larger and learned by the model.

The only idea we care about right now is this:

different Query-Key pairs produce different scores.

A larger score means the two vectors are more aligned in the learned space, so the attention mechanism can treat that token as more relevant to the current Query.

So for our sentence, we may imagine getting something like:

bank → money : 7.2

bank → deposited : 5.8

bank → I : 1.1

bank → the : 0.4

These are completely made-up numbers, but now we have something useful.

The model has converted the vague question:

"Which words matter for bank?"

into actual numerical scores.

And this is a really important step.

Because attention is not the model magically "understanding relationships". At the end of the day, it has to turn those relationships into numbers that can be used in computation.

That is what these Query-Key scores are doing.

Now let's switch the sentence:

"We sat on the bank of the river."

The token is still bank.

But the surrounding context has changed.

So the representations going into this attention layer are different, which means the resulting Queries and Keys can also behave differently.

Now perhaps river gets a very high score with bank, while something like sat gets a smaller one.

So maybe:

bank → river : 8.4

bank → sat : 2.7

bank → the : 0.6

Again, same token bank.

But because the context is different, the relationships that matter can also become different.

And this is exactly what we wanted from the beginning.

We didn't want a fixed rule saying:

"bank should always look at money."

That would obviously fail.

We wanted the model to learn, from the current context, which tokens are useful right now.

There is one more thing here that is worth clearing up. You may be thinking:

Why do we need separate Query and Key vectors at all?

Why don't we simply compare the token embeddings directly?

This is actually a very good question.

And the reason is that the representation a token carries and the representation we want to use for matching relevance don't necessarily have to be the same thing.

A token may contain a lot of information in its hidden state, but for attention we want the model to learn a specific way of asking:

What am I looking for?

and a specific way of advertising:

What information do I have?

So the model learns two separate transformations.

From the current token representation, one transformation creates the Query, and another creates the Key.

If the token representation is x, then very roughly:

where W_Q and W_K are learned matrices.

So Query and Key are not some extra information that came from outside.

They are simply different learned views of the same token representation.

One view is useful for asking.

One view is useful for matching.

And by comparing them, we get our attention score.

Okay, now we have solved one part of the problem.

We know which tokens are relevant.

But we still haven't actually taken any information from them.

Suppose money receives the highest attention score for bank.

Great.

But what exactly do we take from money?

Its Query?

Its Key?

Its entire hidden representation?

This is where the third part of attention comes in:

Value.

---

Okay, so now we have reached an interesting point.

For the token bank, we have already calculated how relevant the other tokens are. Maybe something like:

money → 7.2

deposited → 5.8

I → 1.1

the → 0.4

Great.

But these are just scores. Knowing that money is important doesn't actually give bank any information from money. We still need something that answers:

"Okay, this token is important. Now what information should I actually take from it?"

And this is where the third vector comes into the picture:

Value.

You can think of it like this:

Query and Key help us figure out where to look.

Value gives us what to take from there.

So every token has all three:

1. Query Q
2. Key K
3. Value V

For money, its Key may help the model decide whether money is relevant to bank.

And once we decide that it is relevant, the Value of money contains the information that can actually contribute to the new representation of bank.

Again, don't take the words Query, Key, and Value too literally. They are all just vectors created through learned transformations.

Very roughly:

Same token representation x, but three different learned projections, because all three are doing different jobs. Now suppose our relevance scores for bank look like this:

money → 7.2

deposited → 5.8

I → 1.1

the → 0.4

We somehow want bank to take a lot of information from money, a good amount from deposited, and much less from I or the.

But directly using numbers like 7.2, 5.8, 1.1 is a bit awkward.

What does 7.2 even mean as an amount of information?

It would be much easier if we could convert these scores into something like:

money → 0.55

deposited → 0.32

I → 0.08

the → 0.05

Now this looks much more useful.

You can almost read it as:

Take around 55% of the attention from money, 32% from deposited, and much smaller amounts from the others.

This is roughly where softmax comes in.

Softmax takes all these raw attention scores and converts them into positive weights that add up to 1.

So our raw Query-Key scores:

become some set of normalized attention weights.

Don't care about the exact numbers here. The important thing is that now we have a clean answer to:

"How much should I take from each token?"

And then we finally use the Values.

Suppose the attention weight for money is large. Then:

will contribute a lot.

If the weight for the is tiny, then:

will contribute very little.

We do this for all the tokens and add everything together. So very roughly:

And this resulting vector becomes the attention output for bank.

Now stop here for a second, because this is basically the whole core idea of attention.

We started with a representation of bank. Then we asked:

Which other tokens are relevant to me?

Query and Key helped answer that. Then:

How much should each one matter?

Softmax turned those scores into usable weights.

And finally:

What information should I actually take from those tokens?

That came from the Values.

So the complete flow is basically:

Query + Key → relevance scores

Softmax → attention weights

Attention weights + Values → new contextual information

That is attention at its core.

Let's connect this back to our original example once more.

We have:

"I deposited some money in the bank."

Before attention, bank has some representation.

After attention, its representation can now contain useful information coming from tokens such as money and deposited.

Now take:

"We sat on the bank of the river."

The attention pattern can be different.

Here, river may receive a much larger attention weight.

So the information flowing into the representation of bank is different.

This is how the same token can end up with different contextual representations depending on where it appears.

And I think this is the point where the word attention itself starts feeling quite natural.

The token is literally assigning different amounts of importance to different parts of the context and collecting information accordingly.

Now, if you have seen the famous attention equation before:

now it should already look much less random than it probably did before.

Because almost everything in that equation is something we have already derived naturally.

QKᵀ

is basically:

Compare Queries with Keys and get relevance scores.

softmax(...)

is:

Turn those raw scores into attention weights.

And then:

...V

means:

Use those weights to combine the Values.

There is only one weird-looking part left:

√dₖ

Why are we dividing by that?

We will come to it.

But before going deeper into the exact equation, I think there is one more thing we should understand first.

Until now, we have been thinking about attention from the perspective of one token.

We took bank and asked how it interacts with all the other tokens.

But obviously bank is not the only token doing this.

money is also attending.

deposited is also attending.

Every token in the sequence has its own Query and is looking at the Keys of the other tokens.

So what happens when every token attends to every other token at the same time?

That is where the matrix form of attention starts appearing. And that is also where the famous S × S attention matrix comes from.

---

Okay, until now we have been looking at attention from the perspective of one token.

For bank, we took its Query and compared it with the Keys of all the other tokens. That gave us the relevance scores, then softmax converted those scores into attention weights, and finally we used those weights to combine the Values.

But obviously bank is not the only token doing this.

Every token has its own Query.

So if our sentence contains S tokens, we basically have S Queries, S Keys, and S Values.

Instead of calculating everything token by token, we put all the Query vectors together into one matrix Q, all the Keys into another matrix K, and all the Values into a matrix V.

If each Query and Key has dimension d_k, then:

Q has shape S × d_k

K has shape S × d_k

Now remember what we wanted to do.

Every Query needs to be compared with every Key.

So instead of manually doing:

Q₁ · K₁

Q₁ · K₂

Q₁ · K₃

and then doing the same thing again for Q₂, Q₃, and so on, we can do all of these comparisons together with one matrix multiplication:

Why Kᵀ?

Because Q is:

while Kᵀ becomes:

So:

And now we get something very important:

an S × S attention-score matrix.

Let's make this less mathematical for a second.

Suppose our sequence has only four tokens:

I deposited money bank

Then the attention-score matrix is basically storing something like:

Each row belongs to one Query token.

And every column represents a Key token that it can look at.

So the bank row is basically answering:

How relevant is I to me? How relevant is deposited? How relevant is money? How relevant is bank itself?

And the money row is asking the same kind of question from the perspective of money.

This is what I think is useful to actually visualize.

QKᵀ is not some mysterious matrix multiplication added because the transformer paper needed more maths :)

It is literally calculating all Query-Key relevance scores together.

Now we still have those raw scores.

So, just like we did earlier for one token, we apply softmax.

But here softmax is applied row by row.

Why row by row?

Because every row belongs to one Query token, and we want the attention weights for that token to add up to 1.

So maybe after softmax, the row for bank looks something like:

I → 0.03

deposited → 0.27

money → 0.61

bank → 0.09

while the row for money may look completely different.

Each token gets its own distribution over the sequence.

Now we have an entire matrix of attention weights.

And after that we multiply this matrix with V.

So the complete thing starts looking like:

Every row of the attention matrix tells us how much of each Value vector should be taken for that particular token.

And once we multiply by V, we get the new contextual output for all the tokens together.

So what we previously understood for one token:

compare with everyone → get weights → combine Values

is now simply happening for the whole sequence at once using matrix operations.

There is one piece I intentionally left out from the equation until now.

The actual attention equation is:

We already understand almost everything here.

QKᵀ gives us the Query-Key scores.

Softmax converts them into attention weights.

Multiplying by V gathers the actual information.

The only thing left is:

Why are we dividing by √d_k?

Suppose the dimensions of our Query and Key vectors become large.

A dot product is basically a sum of many multiplications. So as the number of dimensions grows, the magnitude of these dot-product scores can also become quite large.

If these numbers become very large before going into softmax, softmax can become extremely sharp. One token may get something extremely close to 1, while almost everything else becomes close to 0.

That can make training unstable or harder.

So we scale the dot-product scores down by:

where d_k is simply the dimension of the Key/Query vectors.

That's where the name Scaled Dot-Product Attention comes from.

We are doing a dot product, but scaling it before softmax.

Now something else should start becoming visible from the matrix itself.

Remember:

QKᵀ

produced an:

S × S matrix.

So if we have:

1,000 tokens,

there can be roughly:

1,000 × 1,000 = 1,000,000

Query-Key relationships.

For 10,000 tokens:

10,000 × 10,000 = 100,000,000

And if the sequence length doubles, the number of Query-Key pairs doesn't just double.

It becomes roughly four times larger.

In mathematical terms, this part grows quadratically with sequence length:

And this is one of the reasons attention becomes such a big problem when context lengths get very large.

With a short sequence, this is manageable.

But once we start talking about tens of thousands, hundreds of thousands, or even millions of tokens, an S × S interaction pattern becomes very expensive.

And now things like FlashAttention, sparse attention, sliding-window attention, etc. start making a little more sense.

People are not optimizing attention just because attention happens to be popular.

The basic operation itself can become very expensive as the sequence gets longer.

But before going into any of those optimizations, there is still one very important thing missing from our current attention matrix.

Right now, every token can attend to every other token.

For a decoder-only autoregressive LLM, that would create a very obvious problem.

A token could look into the future.

And as we already discussed in the prefill/decode article, that would basically let the model cheat.

So next we need to understand causal masking.

---

Okay, so right now we have an attention matrix of size S × S.

Every row belongs to one token, and every column represents another token it could potentially attend to.

But there is a problem.

Suppose our sequence is:

"The cat sat on the mat."

Imagine we are currently at the token:

cat

Should cat be allowed to attend to mat?

No.

Because mat comes later in the sequence.

And remember, we are talking about a decoder-only autoregressive model. The basic rule is very simple:

while predicting the next token, the model should only have access to the tokens that came before it.

It should not be allowed to look into the future.

Otherwise, we are basically giving it the answer.

Let's take an even simpler example.

Suppose the sequence is:

"The capital of France is Paris."

Imagine the model is processing:

"The capital of France is"

and trying to learn how to predict the next token.

If the token is could already attend to Paris, then what exactly are we asking the model to learn?

The answer is sitting right there.

It is almost like giving someone this exam:

The capital of France is ______

while Paris is already written next to the question :)

So we need some way to prevent each token from seeing tokens that appear after it.

That is what the causal mask does.

Let's say we have four tokens:

A B C D

Without any mask, the attention-score matrix conceptually looks like this:

Every token can look at every other token.

But for causal attention, what we actually want is:

A can only see itself.

B can see A and B.

C can see A, B, and C.

D can see everything before it, including itself.

So the visible part of the matrix forms this lower-triangular shape.

This is why, when you see visualizations of causal attention, you will often see half of the attention matrix blocked out.

Now how do we actually "block" those positions mathematically?

We already have our attention scores:

Before applying softmax, we add a mask to these scores.

For positions that are allowed, nothing changes.

For future positions, we put something effectively like:

Why -infinity?

Because after softmax:

So those future tokens receive an attention weight of exactly 0.

The model simply cannot take information from them.

If we include the mask in the full equation, you may see attention written like:

where M is the causal mask.

You don't really need to memorize this new version separately. The only thing happening is:

calculate attention scores → block future positions → softmax → combine Values.

Now there is something here that initially feels slightly confusing.

During prefill, the complete prompt is already available.

So if the model already has all the prompt tokens, why can't token 2 just look at token 10?

Because "available in memory" and "allowed to use as information" are two completely different things.

We want to process all the prompt tokens in parallel because that makes the computation much faster.

But we still need to preserve the autoregressive rule.

So during prefill, we basically get both:

parallel computation

and

causal information flow.

All the token positions can be computed together, but the causal mask makes sure that each position only uses information from the past.

This is a really important distinction.

The model does not have to process the prompt token by token just because it is autoregressive.

It can process the prompt in parallel.

It just cannot let one token cheat by taking information from future tokens.

During decode, the situation is slightly different.

Suppose we already have:

"The capital of France is Paris"

and now we want to generate the next token.

There are no future generated tokens yet.

We only have the current context.

So the newest token can attend to all the tokens that already exist.

This is also why decode feels different from prefill.

During prefill, we have many known token positions at once, so the causal mask is important to stop information from leaking between future positions.

During normal autoregressive decode, we are only generating the next token from what already exists.

There is simply no future output token available to look at yet.

So now we have a much more complete picture of one attention operation.

For every token:

1. Create a Query, Key, and Value.
2. Compare the Query against Keys to get relevance scores.
3. Scale those scores.
4. Block future positions using the causal mask.
5. Apply softmax to get attention weights.
6. Use those weights to combine the Values.

And that gives us contextual information for the token.

At this point, basic scaled dot-product causal self-attention is mostly there.

But there is still one thing we have been simplifying.

So far, we have acted as if every token has only one Query, one Key, and one Value and therefore only one way of looking at the context.

Modern Transformers don't usually do that.

They let the model look at the same sequence in several different ways at the same time.

And that brings us to multi-head attention (MHA).

---

So until now, we have basically understood one attention operation.

A token creates a Query, Key, and Value. The Query compares with the Keys, we get attention scores, apply softmax, combine the Values, and finally get a new contextual representation.

But there is one limitation in the way we have been thinking about it.

We are assuming there is only one attention pattern.

Let's take a sentence:

"The animal could not cross the street because it was too tired."

When the model is processing the token it, there can be many different relationships worth understanding.

One part of the model may care about:

Which noun does it refer to?

Another may care about the grammatical structure of the sentence.

Another may care about nearby tokens.

Another may care about some longer-range semantic relationship.

Trying to force all of these into one single attention pattern is probably not ideal. So instead of having one attention operation, Transformers use multiple attention heads.

This is what we call Multi-Head Attention.

The easiest way to think about it is that the model gets multiple different ways of looking at the same sequence.

One head may learn to focus more on certain kinds of relationships, another head may focus on something else, and so on.

I would not take this too literally and say:

head 1 understands grammar head 2 understands names head 3 understands pronouns

Real models are obviously much messier than that.

But the intuition is that multiple heads allow the model to capture different relationships between tokens at the same time.

Now mathematically, what changes?

Instead of having one big Query, Key, and Value space, we split the hidden dimension across multiple heads.

So if the model hidden dimension is D and we have H heads, then each head may work with roughly:

Each head has its own Query, Key, and Value projections.

So head 1 performs its own attention.

Head 2 performs its own attention.

Head 3 performs its own attention.

And so on.

Each head produces an output, and at the end we concatenate the outputs from all the heads and project them back into the model dimension.

So very roughly:

same sequence → multiple attention views → combine them together

That is Multi-Head Attention.

And honestly, for the basic understanding, this is enough.

You don't need to make it more complicated than this initially.

Now if you have been reading about modern LLM architectures, you may have seen terms like:

MHA, MQA, GQA

and maybe wondered what exactly changed.

Very roughly:

MHA - Multi-Head Attention

This is what we just discussed.

Every attention head has its own Query, Key, and Value.

So if we have 32 heads, we may effectively have 32 sets of Q, K, and V heads.

This works well, but there is one problem that becomes especially important during inference.

Remember the KV cache?

We need to store Keys and Values for previous tokens.

If every head has its own K and V, then the KV cache becomes large.

And for long contexts or many concurrent requests, that becomes expensive.

So people started reducing the number of K/V heads.

---

MQA [Multi-Query Attention]

In Multi-Query Attention, we still have multiple Query heads.

But instead of every Query head having its own Key and Value head, all Query heads share the same K and V.

So conceptually:

many Q heads → one shared K head + one shared V head

This dramatically reduces the KV cache size and also reduces the amount of K/V data that needs to be read during decode.

Which is very useful for inference.

The trade-off is that we are sharing more information, so we lose some flexibility compared to having completely separate K/V heads for every Query head.

---

GQA [Grouped-Query Attention]

GQA is basically the middle ground between MHA and MQA.

Instead of:

one K/V for every Query head

or:

one K/V shared by all Query heads

we create groups.

For example, maybe 8 Query heads share one K/V head.

So:

many Query heads → fewer K/V heads

This gives us a nice balance.

We keep more flexibility than MQA, but we reduce KV-cache size and memory bandwidth compared to full MHA.

This is why GQA has become very common in modern LLMs.

So if you want one simple picture:

• MHA: every Q head has its own K/V
• MQA: all Q heads share one K/V
• GQA: groups of Q heads share K/V

That's basically the core idea.

---

And I think this is a good place to stop. Because from here, attention starts branching into a lot of different directions:

FlashAttention.

PagedAttention.

Sliding-window attention.

Sparse attention.

MLA.

Different KV-cache optimizations.

And many more.

All of these are interesting, but if I try to put everything into this article, then it may become an book, instead of article.

The goal here was simply to make attention itself feel natural.

Why do we need it?

How does a token get context from other tokens?

Why do we have Query, Key, and Value?

What is the attention matrix?

Why do we need causal masking?

And why do we use multiple heads?

If these things are clear, then most of the advanced attention ideas become much easier to understand later.

And if u want, I can write the next one specifically on things like FlashAttention, PagedAttention, GQA/MQA in more depth, MLA, DSA and how attention changes during inference etc. LMK in replies/DM.

Besides, this was a pretty long & effortive for me, and hopefully for you too :)

But good things take time, and I wanted to make the understanding of attention good, complete, and correct. Hopefully u found it worth your time.

See u next one, bye bye.