The confident machine, and what that confidence is really worth

Every so often somebody forwards me a screenshot of a chatbot getting something spectacularly wrong, usually with a note that says so much for artificial intelligence. I understand the reflex. What I want to explain is why the machine sounded so sure while it was being so wrong, because once you see how the sureness is made, it stops being unsettling and starts being useful.

Sure is not the same as right

When one of these systems tells you it is confident, it is not making you a promise. It is reporting a number it worked out about its own guess, and that number was learned from examples, not from any contact with the thing you are asking about. A weather app that says seventy per cent chance of rain is not seventy per cent certain it will rain on you. It is saying that on days that looked like today, it rained about seven times in ten. The honest version of confidence is always a statement about the past dressed up as a statement about the future.

Where the number comes from

Think of the old game of twenty questions. You narrow the field with each answer: animal, larger than a cat, kept indoors. A machine does something like this at enormous speed, except that instead of asking clean yes or no questions it weighs thousands of small hints at once, each one nudging the answer a little. Add all the nudges up and you get a score. Turn the score into a percentage and you have your confidence. Nothing in that process checks whether the answer is true. It only checks whether the answer fits the pattern of answers the machine has seen rewarded before.

The part that gets left out

Here is the bit that rarely makes the headlines. These systems are often most confident exactly where they are most likely to be wrong, because a wrong answer that looks like a hundred right ones will carry the same high score. A spam filter trained on a decade of junk mail will wave through a brand new kind of scam with complete serenity, precisely because it has never seen one. The confidence was real. The scam was also real. The two had nothing to do with each other.

A machine is never more dangerous than when it is fluent about something it has never met.

What to do with all this

None of this means the number is useless. A high confidence score across thousands of ordinary cases is genuinely worth having, in the way that a good weather forecast is worth having even though it is wrong sometimes. The trick is to treat confidence as a weather report and not as a witness statement. Ask what the machine was trained on. Ask whether your question looks like the ordinary cases or the strange ones. And keep a person in the loop wherever being wrong would actually cost you something.

I am not trying to talk you out of using these tools. I use them every day. I am trying to talk you out of the one belief that gets people into trouble, which is that a machine sounding certain has any bearing on whether it is correct. The certainty is arithmetic done very fast. Whether it is right is a separate question, and it is still, for now, one we have to ask ourselves.