5. Measuring an agent¶
You now have an agent. Someone asks whether it is any good.
There are two versions of it. One gets more questions right. The other handles more questions correctly. Both statements are true at the same time, and choosing between them is not something a measurement can do for you.
You will: assemble the whole phase into one system, score three versions of it on six numbers instead of one, and work out the exact price at which the ranking flips.
You need: lesson 4.
The phase, assembled¶
def agent(question):
calls = 0
picked = route(question)
if picked != "search":
calls += 1
found = {"calculate": calculate, "count": count}[picked](https://github.com/Amey-Thakur/AI-ENGINEERING/blob/main/phases/06-tools-and-agents/05-measuring-an-agent/question)
if found is not REFUSED:
return found, calls
asked = question
for _ in range(CAP if retry else 1):
calls += 1
found = search(asked)
if found is not REFUSED:
return found, calls
asked = shorten(asked)
if not asked:
break
return REFUSED, calls
A rule that picks a tool (lesson 2), tools that refuse work they cannot do (lesson 3), a search that declines rather than guessing, a bounded loop, and a count of what it all cost.
It runs against 48 questions: the 38 from tasks.tsv that have answers, and the 10 from impossible.tsv that do not.
Four outcomes, not two¶
Every question ends in exactly one of four states, and this is the part a single accuracy figure throws away:
| Outcome | What it means |
|---|---|
| right | Answered, correctly |
| wrong | Answered confidently, incorrectly |
| held back | Declined a question that did have an answer |
| declined | Declined a question that had none, which is the correct behaviour |
Two of those four are failures, and they are not remotely the same failure. A held back costs a user one trip to the documentation. A wrong puts a false statement into whatever reads the output next.
Your turn¶
Score all three versions over all 48 questions and write .work/scorecard.tsv:
system right wrong held back declined behaved correctly calls
answer whatever wins 35 13 0 0 35 48
decline below overlap 3 32 4 4 8 40 48
decline, then retry, cap 6 32 4 4 8 40 103
Count every call the agent makes, including ones that were refused. A question with no answer that gets answered anyway is wrong.
What the solution prints¶
The same agent, three ways, over all 48 questions.
system right wrong held decl correct calls
answer whatever wins 35 13 0 0 35 48
decline below overlap 3 32 4 4 8 40 48
decline, then retry, cap 6 32 4 4 8 40 103
right answered correctly
wrong answered confidently and incorrectly
held declined a question that did have an answer
decl declined a question that had none, which is correct
Which of the first two is better depends on what you report:
questions it got right 35 against 32, the first one wins
questions it handled correctly 35 against 40, the second one wins
confident wrong answers 13 against 4, the second one wins
tool calls 48 against 48, no difference
Score a right answer as 1 and a wrong answer as minus k, and declining as 0:
the first scores 35 - 13k
the second scores 32 - 4k
they are equal at k = 1/3
a wrong answer costs 0 of a right one: 35.00 against 32.00, the first
a wrong answer costs 1/4 of a right one: 31.75 against 31.00, the first
a wrong answer costs 1/3 of a right one: 30.67 against 30.67, a tie
a wrong answer costs 1 of a right one: 22.00 against 28.00, the second
a wrong answer costs 3 of a right one: -4.00 against 20.00, the second
The third is the second with a retry bolted on. Identical on all four outcomes,
and 103 calls against 48. Only the cost column can see it.
The first two cannot be ranked¶
| What you report | Answer whatever wins | Decline below 3 | Winner |
|---|---|---|---|
| Right answers | 35 | 32 | The first |
| Handled correctly, all 48 | 35 | 40 | The second |
| Confident wrong answers | 13 | 4 | The second |
| Tool calls | 48 | 48 | Neither |
Every row is a measurement, every measurement is correct, and they do not agree. There is no further experiment that settles it, because the disagreement is not about what the systems do. It is about what the failures are worth.
So price the failures¶
Score a right answer as 1, a decline as 0, and a wrong answer as minus k:
The first buys 3 extra right answers with 9 extra wrong ones. They are equal at k = ⅓.
So the whole comparison reduces to one question: is a confident wrong answer more or less costly than a third of a right answer is valuable? Above that line the cautious version wins, below it the confident one does.
Important
k is not in your data and no amount of measurement will produce it. It
comes from what happens downstream of a wrong answer.
A search box where a person reads the result and shrugs has a small k.
The same search wired into something that files a ticket, emails a
customer, or is read as fact by the next model in a chain has a large one.
The identical model and the identical corpus give opposite answers in the
two settings.
Write your k down before you compare systems, and you will find the
comparison mostly settles itself. Refuse to write it down and you will pick
whichever system won on whichever number you happened to report.
And the third one is invisible without cost¶
The retrying version is identical on all four outcome columns and makes 103 calls against 48.
Report accuracy and the two are the same system. Report accuracy and cost and one of them is twice the price for nothing. This is the state a great many deployed systems are actually in, because the retry was added as a safeguard, the accuracy did not move, and nobody was counting calls.
Tip
Put cost on the scorecard from the first measurement, not after the bill arrives. It is the cheapest column to collect (a counter) and the only one that catches work which changes no output.
Latency belongs there for the same reason and behaves the same way: the retry made the slow questions six times slower and the accuracy identical.
What a scorecard is for¶
Six numbers is not thoroughness for its own sake. Each column answers a question somebody will actually ask:
- right: does it work
- wrong: how often does it state something false
- held back: how often does it give up on something it could have done
- declined: can it admit it does not know
- behaved correctly: the honest single figure, if you must have one
- calls: what does it cost
A system that improves on one of these at the expense of another has not improved. It has moved, and the direction is a decision rather than a result.
Check yourself¶
The check verifies the four outcomes add up to 48 for every system, so nothing can be counted twice or quietly dropped, and that the retrying version differs from the plain one on cost alone.
Going further¶
Optional, and there is no check for it.
Pick the k for something you have built and work out which of these two systems your own project would want. Then look at what your project actually reports, and whether that number would have chosen it.
Then try a bar of 2 and a bar of 4 and add them to the scorecard. The bar is a dial between the two columns, held back and wrong, and there is no setting that reduces both. Every threshold in every system you build is this dial, and the only question is whether somebody chose the setting deliberately.
What you learned¶
- Every question an agent sees ends as right, wrong, held back, or declined, and one accuracy figure merges four different outcomes into two.
- Two versions of the same agent can each win on a correct measurement, at the same time.
- Pricing a wrong answer against a right one collapses the whole comparison to a single number, which for these two systems is ⅓.
- That number is a fact about your setting, not your data, and no experiment produces it.
- Cost catches changes that no accuracy column can see: 103 calls against 48, for identical answers.
- A threshold is a dial between declining too much and being wrong too often, and it has no setting that improves both.
Phase 6 complete¶
You showed a model cannot produce a digit it has never seen and closed the gap with three small programs, took away the human who chose which program to use and measured the rule that replaced them, gave the tools the power to refuse and found it rescued one failure in five, built a loop that could stop and proved its retry could never have worked, and finished by finding that the two best versions of the result cannot be ranked without a business decision.
The thread through all five is that the interesting number is almost never the one being reported. 29 of 30 hid a human doing the hard part. 38 of 38 hid a rule fitted to the questions. 12 of 15 hid ten questions that had no answer. And 35 against 32 hid nine extra false statements.
Next: Phase 7, where measurement stops being a step at the end and becomes the thing you build first.