3. Making it talk¶
A model that predicts the next word can write.
Ask it for a word, add that word to what you have, ask again. That loop is how every model you have used produces text, and you are about to watch its weaknesses in the open, where a larger model would hide them.
You will: generate sentences three ways, and see exactly what a memory of one word can and cannot do.
You need: lesson 2.
Always take the likeliest word¶
The obvious approach. At each step pick whatever the model thinks most likely.
It produces, in full:
the incident
Two words. The commonest word to start a sentence is the, the commonest word after the is incident, and the commonest thing after incident is the end of the sentence. So it stops.
Important
This is greedy decoding, and this is its failure mode at every scale. It is deterministic, so it produces exactly one output forever, and it walks straight into whatever local rut is most common. Large models greedily decoded produce bland text and repeat themselves for the same reason: at every step the safest word is chosen, and safe words lead to safe words.
Draw at random instead¶
Rather than taking the likeliest, draw in proportion. A word seen ten times is ten times as likely to be drawn as one seen once, but the rare one can still happen.
Now the model produces different text each time, and the text has some life in it.
The dial between them¶
Reshape the weights before drawing:
Temperature is that dial.
| Temperature | What it does |
|---|---|
| Below 1 | Sharpens the gap. Likely words get likelier, towards greedy. |
| Exactly 1 | The counts as they are. |
| Above 1 | Flattens it. Rare words get a real chance. |
This is the same temperature setting exposed by every model API you will ever use, and it is doing exactly this.
Note
The seed is fixed so your run matches the page. Real generation uses a random seed, which is why the same prompt gives different answers twice. Fixing the seed is also how you reproduce a bug in generated output, and it is the first thing to do when one is reported.
Your turn¶
Generate one greedy sentence and several sampled ones at different temperatures. Write .work/sentences.tsv:
Tip
The check does not compare your text to anyone's, because sampling makes that meaningless. It verifies the greedy sentence exactly, and then verifies that every adjacent pair in every sentence you produced really occurs in the corpus. That is the difference between text generated by the model and text from anywhere else.
What the solution prints¶
Always taking the likeliest word:
the incident
Temperature 0.4, sharpened: the model plays safe:
the issue were never on the code
the readme explains how to the model because the ticket was scoped to fix forward
the customer
the incident
the meeting and the incident
Temperature 1, the counts as they are:
we shipped the readme explains how to do
the simpler approach and the accident
the live traffic to do
the database connection dropped when the call from another account was confident and rebalanced within the cause
assertions should be about the score improved and more layers did not
Temperature 1.6, flattened: rare words get a real chance:
we could explain it builds the author
review and received
monitoring watches the number of an alert was stale
coverage tells you what was told the index
a version at the database restored from its own logs
Sentences produced: 15
Shortest 2 words, longest 17, average 7.7
Read one of them closely¶
the database connection dropped when the call from another account was confident and rebalanced within the cause
Take it two words at a time. the database. database connection. connection dropped. dropped when. Every pair is fine. Several are lifted straight from real sentences.
Now read it as a sentence. A connection dropped when a call was confident, and then rebalanced within the cause. It is nonsense, and it is nonsense in a very specific way: it changes subject roughly every two words.
That is the model's memory, made visible. It knows exactly one word of history. By the time it writes confident it has entirely forgotten that the sentence was about a connection dropping. It cannot be about anything, because being about something requires remembering what you were about.
And the temperature dial does not fix it¶
At 0.4 it plays safe and produces short, dull, repetitive text: the customer, the incident, the meeting and the incident. At 1.6 it produces wilder text that is wrong in more interesting ways.
Neither is more coherent than the other, because coherence is not a sampling problem. No decoding strategy can recover information the model never had.
Important
This is the honest summary of the whole counting approach. More context would help, and lesson 2 showed you cannot have it: counting triples makes most contexts unseen. So the model must generalise across contexts rather than count them, and that means it must know that server and database are similar.
Which is exactly what phase 3 built, and what the next lesson finally puts to use.
Check yourself¶
Going further¶
Optional, and there is no check for it.
Set the temperature to 0.05 and generate ten sentences. They will nearly all be identical, because sharpening far enough turns sampling back into greedy. Then try 5.0 and watch it approach drawing uniformly at random. The dial has both extremes inside it, and the useful range is narrower than it looks.
What you learned¶
- Generating text is predicting the next word, repeatedly, with your own output as input.
- Greedy decoding is deterministic, and walks into the most common rut available.
- Sampling in proportion to the counts gives variety, and temperature sharpens or flattens those proportions.
- This is the same temperature setting every model API exposes.
- A bigram model changes subject every two words, because its memory is exactly one word long.
- No decoding strategy recovers information the model never had.
Next: 4. A model that generalises, where the vectors from phase 3 let the model share what it learns between similar words.