Skip to content

3. Making it talk

A model that predicts the next word can write.

Ask it for a word, add that word to what you have, ask again. That loop is how every model you have used produces text, and you are about to watch its weaknesses in the open, where a larger model would hide them.

You will: generate sentences three ways, and see exactly what a memory of one word can and cannot do.

You need: lesson 2.


Always take the likeliest word

The obvious approach. At each step pick whatever the model thinks most likely.

word = min(following[word].items(), key=lambda pair: (-pair[1], pair[0]))[0]

It produces, in full:

the incident

Two words. The commonest word to start a sentence is the, the commonest word after the is incident, and the commonest thing after incident is the end of the sentence. So it stops.

Important

This is greedy decoding, and this is its failure mode at every scale. It is deterministic, so it produces exactly one output forever, and it walks straight into whatever local rut is most common. Large models greedily decoded produce bland text and repeat themselves for the same reason: at every step the safest word is chosen, and safe words lead to safe words.


Draw at random instead

Rather than taking the likeliest, draw in proportion. A word seen ten times is ten times as likely to be drawn as one seen once, but the rare one can still happen.

word = generator.choices(choices, weights=weights)[0]

Now the model produces different text each time, and the text has some life in it.


The dial between them

Reshape the weights before drawing:

weights = [count ** (1.0 / temperature) for count in counts]

Temperature is that dial.

Temperature What it does
Below 1 Sharpens the gap. Likely words get likelier, towards greedy.
Exactly 1 The counts as they are.
Above 1 Flattens it. Rare words get a real chance.

This is the same temperature setting exposed by every model API you will ever use, and it is doing exactly this.

Note

The seed is fixed so your run matches the page. Real generation uses a random seed, which is why the same prompt gives different answers twice. Fixing the seed is also how you reproduce a bug in generated output, and it is the first thing to do when one is reported.


Your turn

Generate one greedy sentence and several sampled ones at different temperatures. Write .work/sentences.tsv:

method            sentence
greedy            the incident
temperature 1     the live traffic to do

Tip

The check does not compare your text to anyone's, because sampling makes that meaningless. It verifies the greedy sentence exactly, and then verifies that every adjacent pair in every sentence you produced really occurs in the corpus. That is the difference between text generated by the model and text from anywhere else.


What the solution prints

Always taking the likeliest word:
  the incident

Temperature 0.4, sharpened: the model plays safe:
  the issue were never on the code
  the readme explains how to the model because the ticket was scoped to fix forward
  the customer
  the incident
  the meeting and the incident

Temperature 1, the counts as they are:
  we shipped the readme explains how to do
  the simpler approach and the accident
  the live traffic to do
  the database connection dropped when the call from another account was confident and rebalanced within the cause
  assertions should be about the score improved and more layers did not

Temperature 1.6, flattened: rare words get a real chance:
  we could explain it builds the author
  review and received
  monitoring watches the number of an alert was stale
  coverage tells you what was told the index
  a version at the database restored from its own logs

Sentences produced: 15
Shortest 2 words, longest 17, average 7.7


Read one of them closely

the database connection dropped when the call from another account was confident and rebalanced within the cause

Take it two words at a time. the database. database connection. connection dropped. dropped when. Every pair is fine. Several are lifted straight from real sentences.

Now read it as a sentence. A connection dropped when a call was confident, and then rebalanced within the cause. It is nonsense, and it is nonsense in a very specific way: it changes subject roughly every two words.

That is the model's memory, made visible. It knows exactly one word of history. By the time it writes confident it has entirely forgotten that the sentence was about a connection dropping. It cannot be about anything, because being about something requires remembering what you were about.


And the temperature dial does not fix it

At 0.4 it plays safe and produces short, dull, repetitive text: the customer, the incident, the meeting and the incident. At 1.6 it produces wilder text that is wrong in more interesting ways.

Neither is more coherent than the other, because coherence is not a sampling problem. No decoding strategy can recover information the model never had.

Important

This is the honest summary of the whole counting approach. More context would help, and lesson 2 showed you cannot have it: counting triples makes most contexts unseen. So the model must generalise across contexts rather than count them, and that means it must know that server and database are similar.

Which is exactly what phase 3 built, and what the next lesson finally puts to use.


Check yourself

python tools/run_lessons.py phases/04-a-language-model-you-wrote-yourself/03-making-it-talk


Going further

Optional, and there is no check for it.

Set the temperature to 0.05 and generate ten sentences. They will nearly all be identical, because sharpening far enough turns sampling back into greedy. Then try 5.0 and watch it approach drawing uniformly at random. The dial has both extremes inside it, and the useful range is narrower than it looks.


What you learned

  • Generating text is predicting the next word, repeatedly, with your own output as input.
  • Greedy decoding is deterministic, and walks into the most common rut available.
  • Sampling in proportion to the counts gives variety, and temperature sharpens or flattens those proportions.
  • This is the same temperature setting every model API exposes.
  • A bigram model changes subject every two words, because its memory is exactly one word long.
  • No decoding strategy recovers information the model never had.

Next: 4. A model that generalises, where the vectors from phase 3 let the model share what it learns between similar words.