Glossary¶
This course teaches the ideas before it teaches their names, and in several places it never uses the standard name at all. Phase 5 measures found-by-k and average reciprocal place. The literature calls those recall@k and mean reciprocal rank. Phase 3 spends a lesson on the company a word keeps, which is the distributional hypothesis. Phase 4 decides when to stop training by holding out data, which is early stopping.
That is a deliberate choice: a name learned before the thing it names is a word you can repeat and cannot use. But it leaves a gap, and this file closes it.
Every row links to the lesson where you built the thing. A definition you have already implemented reads differently from one you have not, and you will have implemented all of these.
Tip
Read this after a phase, not before. The whole design of the course is that the idea arrives first and the vocabulary arrives second, and reading the vocabulary first gives it nothing to attach to.
Phase 0. Your machine¶
| In this course | Also called | What it is |
|---|---|---|
| A window you type into | Terminal, shell, command line, REPL | The program that runs your typed commands. bash and zsh on Linux and macOS, PowerShell on Windows. |
| Where a file actually is | Absolute and relative paths, working directory | The difference between naming a file from the root of the disk and naming it from where you happen to be standing. |
| A copy of Python that is only this project's | Virtual environment, venv, dependency isolation |
A private directory of packages, so two projects can want two versions of the same thing. |
| Saving what you did | Version control, commit, repository, fork | A record of every change, and the ability to return to any of them. |
Phase 1. Teach it to tell two things apart¶
| In this course | Also called | What it is |
|---|---|---|
| Splitting on spaces | Tokenisation, tokeniser | Cutting text into the units a model counts. .split() is the naive one, and the lesson names four things it gets wrong. |
| Counting which words appear | Bag of words, feature vector, feature extraction | Representing a document as counts of its words, discarding order entirely. |
| A rule you wrote by hand | Heuristic, rule-based baseline | A human-written decision rule. In this course it scored 18 of 20 and beat everything learned for three phases. |
| Letting it find the numbers | Perceptron, supervised learning, weights, bias | The 1958 algorithm that adjusts a weight per feature whenever it gets an example wrong. |
| Messages it has never seen | Held-out set, test set, generalisation | Data kept back from training, because a score on data you fitted to is not a score. |
| Getting it right for the wrong reason | Overfitting, error analysis, spurious correlation | Learning a pattern that holds in your sample and not in the world. |
Phase 2. A network you wrote yourself¶
| In this course | Also called | What it is |
|---|---|---|
| The thing one line cannot do | Linear separability, the XOR problem | A dataset no single straight cut can divide, which is why one layer is not enough. |
| A second layer | Hidden layer, multilayer perceptron, MLP, feedforward network | Stacking a transformation before the decision, so the decision can be made in a space where the data is separable. |
| A curve instead of a cliff | Sigmoid, logistic function, activation function | A smooth threshold. A hard one has no useful slope, so nothing can be tuned through it. |
| How wrong it is, as one number | Loss function, objective, cost | The single quantity training tries to reduce. |
| Rise over run, measured by nudging | Gradient, finite differences, numerical gradient | Which way to move each weight, found by changing it slightly and looking, with no calculus. |
| Working out every nudge at once | Backpropagation, the chain rule, reverse-mode autodiff | Computing all the gradients in one backward pass instead of one forward pass per weight. |
| Checking the fast way agrees with the slow way | Gradient checking | Comparing backpropagation against nudging, which is how you find the sign error you would otherwise ship. |
| How big a step to take | Learning rate, step size, hyperparameter | The multiplier on each update, with a wrong answer in both directions. |
Phase 3. Words that know what they mean¶
| In this course | Also called | What it is |
|---|---|---|
| A word the model has never met | Out-of-vocabulary, OOV, the vocabulary problem | A word absent from training, which a count-based model can say nothing about. |
| The company a word keeps | The distributional hypothesis, Firth 1957 | The idea that a word's meaning can be read off the words that surround it. |
| A profile per word | Co-occurrence matrix, context window | A count of how often each word appears near each other word. |
| Comparing direction rather than size | Cosine similarity | The standard way to compare two vectors when their magnitude is not the point. |
| How much more often than chance | Pointwise mutual information, PMI, PPMI | Dividing a co-occurrence count by what chance alone would produce, which is what makes meaning appear. |
| Fewer numbers that say more | Dimensionality reduction, SVD, latent semantic analysis, embeddings | Compressing a sparse profile of hundreds of numbers into a dense handful that carry the same facts. |
| Multiplying and normalising until it settles | Power iteration, eigenvector | Finding the direction the data varies along most, with multiplication and nothing else. |
| Does it help | Extrinsic evaluation, downstream task | Judging a representation by whether it improves the job you actually have, rather than by how good its neighbours look. |
Phase 4. A language model you wrote yourself¶
| In this course | Also called | What it is |
|---|---|---|
| Predicting the next word | Language model, n-gram model, bigram, the Markov assumption | Assigning a probability to what comes next, given what came before. |
| Never say never | Smoothing, add-k, Laplace smoothing, interpolation, backoff | Giving unseen pairs a floor, because one impossible word makes a model infinitely bad. |
| An effective vocabulary size | Perplexity, held-out likelihood | How much probability the model put on what actually happened, read as a number of equally likely options. |
| Making it talk | Decoding, greedy decoding, sampling, temperature, top-k | Turning a distribution over next words into text, and the dial that trades safety for variety. |
| Turning scores into probabilities | Softmax, cross-entropy loss, the log-sum-exp trick | The function that normalises arbitrary scores, and the subtraction that stops it overflowing. |
| Deciding when to stop | Early stopping, validation set, the three-way split | Using a third slice of data to choose when training has gone far enough, because that choice needs its own data. |
| What it knows | Parametric knowledge, memorisation | What a model can state from its weights alone, which turns out to be far less than it has read. |
Phase 5. Finding the right thing to say¶
| In this course | Also called | What it is |
|---|---|---|
| It has read it and cannot recall it | Closed-book and open-book question answering, knowledge-intensive tasks | Answering from the weights against answering from a document you fetched. The gap between them is the reason retrieval exists. |
| Finding it by the words | Lexical retrieval, keyword search, sparse retrieval, BM25, TF-IDF | Ranking documents by the words they share with the question. |
| Finding it by meaning | Semantic search, dense retrieval, vector search, embedding search | Ranking by vector similarity instead of shared words. Measured here at zero on paraphrases. |
| Found-by-k | Recall@k, hit rate at k | Whether the right document is anywhere in the top k, rather than only at the top. |
| Average reciprocal place | Mean reciprocal rank, MRR | One over the position of the right answer, averaged. Rewards putting it first without ignoring second. |
| Retrieve, then answer | Retrieval-augmented generation, RAG, grounding | Fetching a passage and having a model answer from it, rather than from memory. |
| The number nothing downstream can raise | The retrieval ceiling | If search finds the right passage 75% of the time, the honest maximum for the whole system is 75%. |
Phase 6. Tools and agents¶
| In this course | Also called | What it is |
|---|---|---|
| A function the system can call | Tool use, function calling, tool calling | Letting the system run code instead of predicting what the code would have printed. |
| Choosing the tool | Routing, intent classification, tool selection, dispatch | Working out which function a request needs, which is harder than any of the functions. |
| A tool that says no | Input validation, preconditions, guardrails, graceful refusal | Writing down what a tool requires, so a misroute is declined rather than answered. |
| A loop that can stop | Agent loop, ReAct, agentic loop, the control loop | Call a tool, read the result, decide whether to go again. This is what an agent is. |
| Running out of budget | Step cap, max iterations, iteration limit | The bound that decides whether you or the input controls what a request costs. |
| Questions with no answer | Unanswerable questions, abstention, calibrated refusal | The examples no benchmark contains, and the only ones that can measure whether a system can say it does not know. |
| Measuring an agent | Cost-sensitive evaluation, utility-weighted scoring, the precision and recall tradeoff | Scoring several outcomes at once rather than one, and pricing a wrong answer against a right one so that two systems can be ranked at all. |
Phase 7. Measuring whether any of it works¶
| In this course | Also called | What it is |
|---|---|---|
| The number has error bars | Confidence interval, Clopper-Pearson interval, exact binomial interval | The set of true rates that could plausibly have produced your score. Twenty examples buys about twenty points. |
| Is the difference real | McNemar's test, paired significance test, p-value | The right test when two systems ran on the same examples, and far more powerful than comparing intervals. |
| Not enough examples to tell | Statistical power, type II error | Why "no significant difference" and "no difference" are different claims. |
| What an empty function scores | Baseline, majority-class baseline, trivial baseline | The floor any result has to clear before it means anything. On an unbalanced problem it can be above 90%. |
| Deleting a part to see if it mattered | Ablation, ablation study | Removing one component and remeasuring. This course's retrieval half did not survive the test. |
| The test set you keep looking at | Test-set leakage, hyperparameter overfitting, selection bias, optimism | Tuning against the data you then report from, which inflates the number even when the choice is right. |
| A test you can actually run | Regression test, snapshot test, golden file, expectations file | A record of what the system currently does, failures included, so a change reports which examples moved. |
Phase 8. Shipping¶
| In this course | Also called | What it is |
|---|---|---|
| Counting the work | Cost accounting, token accounting, unit economics | Measuring work in units that do not depend on your laptop: tokens, rows examined, documents scanned. |
| The dearest question | Tail latency, p95, p99, the long tail | What the unlucky request costs. The average describes a request that often does not exist. |
| A dictionary the other way round | Inverted index, postings list | Mapping each word to the documents holding it, so a query costs its own words rather than the whole corpus. |
| Words that are nearly everywhere | Stop words, document frequency, IDF | The commonest words, whose removal is a speedup and a change to the system rather than an optimisation. |
| Failing by refusing | Graceful degradation, fail-open, fail-closed, fallback | What a system does when a component is unavailable, and why a fallback is only safe if it can tell it is being misused. |
| The number that noticed | Leading indicator, observability, score distribution monitoring | A signal that moves the moment something breaks, needs no labels, and is free on every request. |
| What it must not say | Data leakage, document-level access control, permission-aware retrieval | Keeping a boundary in a system whose scoring function has no concept of who is asking. |
| The thing you ship | Model card, system card, release criteria, evaluation report | The page of sourced numbers you hand somebody, including the section saying what the numbers do not establish. |
Phase 9. Attention, and the block¶
| In this course | Also called | What it is |
|---|---|---|
| The thing a bigram cannot see | Long-range dependency, subject-verb agreement, targeted syntactic evaluation | A dependency wider than the model's window, which no amount of training or data can close. |
| Looking at every word at once | Attention, scaled dot-product attention, query, key and value | Scoring every earlier position against a query, normalising the scores, and reading a blend of the values. |
| Dividing by the square root | The scaling factor, 1/sqrt(d_k) |
What keeps a dot product of wide vectors from saturating the softmax before training has learned anything. |
| Taking the mechanism out | Ablation, lesion study, knockout | Removing one part and leaving everything else, which is the only way to find out what that part was contributing. |
| An even share | Uniform attention, mean pooling, average pooling | Replacing the learned weights with one over the number of positions, so the model has no query and no key in it at all. |
| Where the signal is | Attention interpretability, faithfulness of attention | The weights say how a blend was mixed, not which ingredient mattered, so a heatmap is not an account of what the model used. |
| Where a word sits | Positional embedding, position encoding | A learned vector per position, added to the word's vector, which is the only way a blend of values can know an order. |
| What the split can answer | Transitive closure, derivable labels, ceiling on a held-back set | The best score anything could get on a held-back half, worked out before the model's score is read. |
| Which examples, not how many | Sample selection, margin, curriculum | The same number of examples teaching everything or nothing, depending on how far apart the things being compared are. |
| Centring and scaling a vector | Layer normalisation, layer norm, LayerNorm | Subtracting the mean and dividing by the deviation, then a learned gain and bias, which keeps a residual stream from compounding. |
| Adding instead of replacing | Residual connection, skip connection, shortcut | Adding a sub-layer's output to its input, which gives the gradient a path that skips the sub-layer entirely. |
| The block | Transformer block, transformer layer, pre-norm block | One attention head with two layer norms, two residual connections and a small feed forward network around it. |
| Checking a gradient by nudging | Gradient checking, finite difference check, numerical gradient | Moving one weight a little and dividing the change in loss by the change in weight, which is the only thing that catches a hand-written backward pass being quietly wrong. |
Words this course does not use, and why¶
A few terms you will meet constantly are absent from the lessons on purpose.
Artificial intelligence. The course says what a thing does instead. Nothing in it becomes clearer for being called intelligent.
Neural. Phase 2 builds a multilayer perceptron and calls it a network of weighted sums, because that is what it is, and the biological word imports a claim the mathematics does not make.
Hallucination. Phase 6 says said something false and counts them. The usual word suggests a malfunction, when producing a fluent wrong answer is the system working exactly as built.
Reasoning. Phase 6 builds a loop that calls a tool and decides whether to go again, and describes it in those words.
Prompt engineering. No lesson needs it, because no lesson calls a hosted model. The habits it stands in for, measuring a change and keeping the examples that showed the problem, are phases 7 and 8.
Where to go after this¶
Every row above is a search term. The course gave you the thing; these give you the literature.
- Retrieval: start at BM25, then Robertson and Zaragoza's survey of the probabilistic relevance framework, which is the method phase 5 lesson 2 measured against.
- Evaluation: Clopper-Pearson (1934) for phase 7 lesson 1, McNemar (1947) for lesson 2. Both are short papers and both are readable.
- Distributional semantics: Firth (1957) for the sentence phase 3 is built on, then PPMI and SVD.
- Agents: read the tool-calling documentation of any provider with phase 6 lesson 3 in mind, and notice how much of it is about what a tool does with input it was never meant to see.
Every entry links to a lesson in this repository, and tools/check_glossary.py fails if one of those links stops resolving.