3. It works without looking¶
The head from lesson 2 clears a ceiling no n-gram can reach, and its attention weights are nearly flat.
On the two sentences it is distinguishing it spreads the weight almost identically: 0.28 / 0.08 against 0.29 / 0.06. The subject, the one word that decides the answer, gets 0.19 on average where spreading the weight evenly would give 0.17.
Something is carrying the signal. On this evidence it is not the weights.
You will: find the signal by taking things away, and finish with a model that has no query and no key in it and scores within one question of the head.
You need: lesson 2.
Three questions, each answered by removing something¶
| Question | What to remove |
|---|---|
| Does the head need its weights? | Train one with the weights nailed to an even share, so there is no query and no key anywhere in it. |
| Does the trained head use its weights? | Force its own weights to an even share at test time, without retraining. |
| Where is the signal, then? | Zero one position's value at a time and see which one matters. |
Both models start from the same seed, so they begin from identical weights and the only difference between them is whether the attention is learned at all.
Your turn¶
Train both, run the ablations, and write .work/located.tsv:
what trained on held back
the attention head 38 16
an even share, trained that way 34 15
the head, weights evened at test 33 15
position 0 silenced 38 15
position 1 silenced 20 10
...
What the solution prints¶
40 sentences to train on, 20 held back. Chance is 10 of 20.
what trained on held back
the attention head 38 of 40 16 of 20
an even share, trained that way 34 of 40 15 of 20
the head, weights evened at test 33 of 40 15 of 20
Does it need the weights? An even share, with no query and no key anywhere in it,
reaches 15 of 20 where the head reaches 16. The learned weights are worth 1.
Does the trained head use them? Evening its own weights at test time costs it
1 of 20. It had barely been relying on them.
So where is the signal? Silence one position's value at a time:
silenced trained on held back
position 0 38 of 40 15 of 20
position 1 20 of 40 10 of 20 <- the subject
position 2 38 of 40 16 of 20
position 3 38 of 40 15 of 20
position 4 36 of 40 15 of 20
position 5 36 of 40 16 of 20
Silencing the subject's value takes it to 10 of 20, which is chance.
Silencing any other position leaves it at 15 or better.
The signal is one vector: the value at the subject's position. The weights decide
how much of it reaches the output, and an even share passes enough of it along.
The weights are worth one question¶
An even share reaches 15 of 20. The learned head reaches 16.
That model has no query matrix and no key matrix. It cannot score a position against anything, because there is nothing to score with. It takes the mean of the values and hands that to the output layer, and it clears lesson 1's ceiling just as decisively.
And the trained head barely uses what it learned: evening its own weights at test time costs it one question. It spent 150 passes learning a query and a key and then produced an answer that hardly depends on them.
Note
One question on twenty is not a difference. Phase 7 lesson 1 puts 16 of 20 at 56.3% to 94.3% and 15 of 20 at 50.9% to 91.3%, and those intervals sit almost on top of each other. The honest statement is that on this task, at this size, a mean beats nothing and attention beats a mean by an amount too small to measure.
And the signal is one vector¶
Silence the subject's value and the model lands on 10 of 20 and 20 of 40. Both are exactly half. Not degraded, not worse: chance, as though the pair had become indistinguishable, which is precisely what happened.
Silence any other position and it loses at most one question.
| Silenced | Held back | |
|---|---|---|
position 0, the |
15 of 20 | |
| position 1, the subject | 10 of 20 | chance |
position 2, that |
16 of 20 | unchanged |
position 3, the |
15 of 20 | |
| position 4, the actor | 15 of 20 | |
| position 5, the verb | 16 of 20 | unchanged |
So the whole mechanism is this: the subject's value vector carries singular or plural, the blend includes a share of it whatever the weights are, and the output layer reads it. The weights decide how much of that vector arrives. An even share passes along a fifth of it, and a fifth is plenty.
Caution
This is why an attention heatmap is not an explanation.
The weights here are a near-uniform smear. A reader shown that picture would conclude the model was using everything a little and nothing in particular, and would be wrong: it is using exactly one position and ignoring the rest. The information about what the model used is in the values, which the picture does not show.
Attention weights tell you how a blend was mixed. They do not tell you which ingredient mattered, because an ingredient with a loud value matters at any mixing ratio above zero. Published attention maps are routinely read as accounts of a model's reasoning, and this is a twelve line experiment showing why that reading does not follow.
What attention is actually for, then¶
Not this task. A dependency that one position decides, in a six word window, with a vocabulary of forty one words, is solved by a mean and a wide enough output layer.
What attention buys is the ability to be selective when selection matters: many positions, most of them irrelevant, and a different one relevant for each query. None of those conditions hold here, which is why the head's weights went flat and nothing was lost by it going flat. A mechanism is not demonstrated by a task that does not need it.
That is worth holding on to the next time a result is attributed to an architecture. The way to find out is the experiment in this lesson: take the mechanism out, leave everything else, and see whether the number moves.
Check yourself¶
The check insists that silencing the subject lands on chance exactly, that no other position costs more than three questions, and that the learned weights stay worth at most two. If the head ever pulls properly ahead of the mean, this lesson is wrong and should be rewritten rather than the check relaxed.
Going further¶
Optional, and there is no check for it.
Silence two positions at once and look for a pair that costs more than the sum of silencing each alone. If none exists, nothing in this model is distributed across positions, which is worth knowing before anybody describes it as building a representation.
Then print the value vectors for server and servers and compare them to a pair that does not differ in number, such as engineer and reviewer. The distance that matters is the one the output layer reads, and you can see it directly.
What you learned¶
- A mechanism's contribution is measured by removing it and leaving everything else, not by looking at what it produced.
- A model with no query and no key, taking the mean of the values, comes within one question of the learned head on this task.
- The trained head barely used its own weights: evening them at test time cost one question.
- Silencing the subject's value lands the model on exactly chance, and silencing any other position costs at most one, so the signal is a single vector.
- Attention weights describe how a blend was mixed, not which ingredient mattered, which is why a heatmap is not an account of what a model used.
- Attention earns its place when selection matters, and nothing here needed selecting.
Next: 4. Where a word sits, where the mean that did so well turns out not to know the order of anything.