1. Predicting the next word¶
Every large language model does one thing: it predicts what comes next.
Everything else, answering questions, writing code, holding a conversation, is that one ability pointed in a direction. You are about to build the same thing, out of counting, in about forty lines.
You will: build a language model, watch it guess the next word, and measure how often it is right on sentences it has never seen.
You need: phase 3.
The whole idea¶
Read a lot of text. Count which word follows which. To predict, look up what usually came next.
That is a bigram model: it looks at exactly one word of context. It is the simplest thing that deserves the name language model, and everything in this phase is a way of making the context longer and the counting smarter.
Marking the edges¶
Two invented tokens. <s> lets the model learn how sentences begin, which is otherwise invisible: without it, the first word of every sentence has no context and the model has nothing to condition on. </s> lets it learn that sentences stop, which matters enormously the moment you generate text, as the next lessons will show.
Measuring it¶
Train on the first 180 sentences of the corpus. Hold back the last 37. Then for every position in a held back sentence, guess the next word and check.
Important
The split is by position rather than random, so everybody gets the same one without needing a seed. And it is by sentence, not by word: splitting mid-sentence would leave half of a held back sentence in the training data, which is the leak from phase 1 lesson 4 wearing a different coat.
Your turn¶
Build the counts and write .work/results.tsv:
Warning
One trap, and it caught me while writing this lesson. following is a
defaultdict, and reading a missing key from a defaultdict creates it.
So checking following[word][next_word] == 0 while measuring quietly inserts
empty contexts into the model you are measuring.
Use following.get(word, {}).get(next_word, 0) instead. The check will tell
you if you did not.
What the solution prints¶
180 sentences to learn from, 37 held back
608 different words, 519 of them with something known to follow
What it expects next:
after 'the' incident (9), customer (8), error (7), meeting (7)
after 'server' cleared (1), doubled (1), ran (1), restarted (1)
after 'we' added (1), cancelled (1), cut (1), deploy (1)
after 'customer' </s> (1), account (1), asked (1), confirmed (1)
at the start of a sentence the (100), a (16), we (10), coverage (3)
Guessing the next word on held back sentences:
right 54 of 391 13.8%
context never seen in training 112
pair never seen in training 302 of 391 77%
What it has learned¶
Read the predictions. After the, it expects incident, customer, error, meeting. At the start of a sentence it expects the, a hundred times over.
Those are real facts about English and about this corpus, learned by nobody explaining anything. The same mechanism at a scale a billion times larger is what makes a modern model seem to know things.
And where it collapses¶
13.8% right. Worse than that, look at the other two numbers.
| Predictions made | 391 |
| Contexts never seen in training | 112 |
| Pairs never seen in training | 302 of 391, or 77% |
For 112 of the 391 predictions the model had never encountered the word it was asked to follow, so it had nothing at all to say.
And 77% of the word pairs in the held back sentences never occurred in training. The model assigns each of those a probability of exactly zero: not "unlikely", but impossible. It is certain that a sentence a human wrote cannot exist.
Important
This is the defining problem of counting based language models, and it does not go away with more data. Language is combinatorial: a vocabulary of 600 words has 360,000 possible pairs, and a real vocabulary of 50,000 has two and a half billion. Most valid pairs will never appear in any corpus, however large, simply because most sentences have never been written.
Counting can only ever tell you about what has already happened. Language is mostly about what has not.
Check yourself¶
python tools/run_lessons.py phases/04-a-language-model-you-wrote-yourself/01-predicting-the-next-word
Going further¶
Optional, and there is no check for it.
Count triples instead of pairs: what follows the server rather than what follows server. The predictions become far more sensible where they exist, and the number of unseen contexts goes up sharply. That trade is the whole story of this phase, and it is why the answer turns out not to be longer counts.
What you learned¶
- A language model predicts the next token, and that single ability underlies all the rest.
- Counting which word follows which is enough to build one.
- Start and end markers let a model learn how sentences begin and stop.
- Split held back data by sentence, or you leak.
- Reading a missing key from a defaultdict creates it, which can corrupt a model while you measure it.
- 77% of held back pairs get probability zero, and that is a property of language rather than of this corpus.
Next: 2. Never say never, where giving everything a small chance turns an impossible model into a usable one.