1. Words into numbers¶
A computer cannot read. It can only count.
Every system you have heard of, every model with a name, is doing arithmetic on numbers that used to be text. The interesting question is not how the arithmetic works. It is how the text became numbers in the first place, and that is a decision a person makes.
You will: read a data file, turn sentences into counts, and find out where the signal in a problem actually lives.
You need: phase 0.
The problem¶
In this folder is messages.tsv: sixty short messages, half of them questions and half statements.
label message
question what time does the meeting start
statement the meeting starts at ten
question how do i reset my password
statement i reset my password this morning
Your job across this phase is to build something that can tell the two apart. Not by asking you. By looking at the words.
Note
The question marks have been removed on purpose. With them the problem is one line of code and teaches nothing. Without them you have to find real signal, which is the situation you are always in with real data.
Reading the file¶
.tsv means tab separated values: columns with a tab between them. It is the plainest useful data format there is, and you can read it with nothing but the open that comes with Python.
with open("messages.tsv", encoding="utf-8") as handle:
next(handle) # skip the header line
for line in handle:
label, message = line.strip().split("\t")
Three things worth naming:
withcloses the file for you when the block ends, including if something goes wrong inside it. Opening a file withoutwithis a habit that costs people later.encoding="utf-8"says how the bytes turn into characters. Leave it out and Python guesses from your operating system, which is why a file that works on your machine breaks on someone else's..strip()removes the newline at the end of the line. Without it your last word on every row isstart\n, notstart, and nothing matches.
Turning a sentence into something countable¶
split() with nothing in the brackets cuts on whitespace. "what time does it start" becomes ["what", "time", "does", "it", "start"].
This is called tokenising, and this is the naive version of it. It is worth knowing now what it gets wrong, because you will fix these later in the course:
| It cannot tell | Example |
|---|---|
| Case | The and the are different words to it |
| Punctuation | start. and start are different words |
| Word forms | run, runs and running share nothing |
| Meaning | quick and fast are as unrelated as quick and tuesday |
Our data is already lowercase with no punctuation, so the first two do not bite today. They will.
Counting, and one decision inside it¶
For each word, count how many messages of each kind contain it.
Look at set(words). A set throws away duplicates, so a word said twice in one message counts once.
That is a real decision and it could go the other way. Counting every occurrence measures enthusiasm; counting messages measures how widespread a word is. For sixty short messages, widespread is the more honest signal, because one message that says "the" four times should not get four votes.
Important
Decisions like this one are the actual work of this field, and they are invisible in the finished product. Nobody writes a paper about whether to use a set. It changes your results anyway.
Your turn¶
Write .work/counts.tsv, with a header row and then one row per word:
Every different word in the messages needs a row. Tabs between the columns.
What the solution prints¶
60 messages: 30 questions, 30 statements
172 different words
Leans towards a question:
does 5 of the questions, 0 of the statements
do 4 of the questions, 0 of the statements
can 4 of the questions, 1 of the statements
how 4 of the questions, 1 of the statements
this 7 of the questions, 4 of the statements
what 4 of the questions, 1 of the statements
Leans towards a statement:
in 0 of the questions, 4 of the statements
minutes 0 of the questions, 2 of the statements
next 0 of the questions, 2 of the statements
takes 0 of the questions, 2 of the statements
the 17 of the questions, 19 of the statements
works 0 of the questions, 2 of the statements
Read that table again¶
The words leaning towards a question are the ones you would have guessed, and that is reassuring rather than boring: the method found, with no help, what a person already knows.
The statement side is more interesting. Those words are not meaningfully about statements. They are just common, and there happen to be slightly more of them on that side. A system built only on the strongest signals would be trusting noise.
Tip
This is the whole shape of the field in one table. Some of your signal is real, some of it is an accident of the data you happened to collect, and nothing in the numbers tells you which is which. Only knowing what the data means does.
Check yourself¶
From the repository root:
The check reads the messages and works the counts out for itself, then compares them with yours. It is not holding a stored answer, so you cannot pass it by copying one.
Going further¶
Optional, and there is no check for it.
Count every occurrence instead of one per message, by dropping the set(). Which words move? Nothing about the data changed, only what you decided to measure, and the answer is different. Keep that feeling.
What you learned¶
- Text becomes numbers by a decision somebody makes, not by magic.
with open(..., encoding="utf-8")is how you read a file safely and portably..split()is the naive tokeniser, and you now know four things it gets wrong.- Counting messages rather than occurrences is a choice that changes the answer.
- Signal and coincidence look identical in a table of counts.
Next: 2. A rule by hand, where you use these counts to make an actual decision, and find out how often you are right.