Last week, I mentioned that Dylan Matthews' suggestion that maybe there was only 10^-67 chance you could affect AI risk was stupendously overconfident. I mentioned that was thousands of lower than than the chance,
per second, of getting simultaneously hit by a tornado, meteor, and al-Qaeda bomb, while
also winning the lottery twice in a row. Unless you're comfortable with that level of improbability, you should stop using numbers like 10^-67.
But maybe it sounds like "one in a million" is much safer. That's only 10^-6, after all, way below the tornado-meteor-terrorist-double-lottery range…
So let's talk about overconfidence.
Nearly everyone is very very very overconfident. We know this from
experiments where people answer true/false trivia questions, then are asked to state how confident they are in their answer. If people's confidence was well-calibrated, someone who said they were 99% confident (ie only 1% chance they're wrong) would get the question wrong only 1% of the time. In fact, people who say they are 99% confident get the question wrong about 20% of the time.
It gets worse. People who say there's only a 1 in 100,000 chance they're wrong? Wrong 15% of the time. One in a million? Wrong 5% of the time. They're not just overconfident, they are
fifty thousand times as confident as they should be.
This is not just a methodological issue. Test confidence in some other clever way, and you get the same picture. For example,
one experiment asked people how many numbers there were in the Boston phone book. They were instructed to set a range, such that the true number would be in their range 98% of the time (ie they would only be wrong 2% of the time). In fact, they were wrong 40% of the time. Twenty times too confident! What do you want to bet that if they'd been asked for a range so wide there was only a one in a million chance they'd be wrong, at least five percent of them would have bungled it?
Yet some people think they can predict the future course of AI with one in a million accuracy!
Imagine if every time you said you were sure of something to the level of 999,999/1 million, and you were right, the Probability Gods gave you a dollar. Every time you said this and you were wrong, you lost $1 million (if you don't have the cash on hand, the Probability Gods offer a generous payment plan at low interest). You might feel like getting some free cash for the parking meter by uttering statements like "The sun will rise in the east tomorrow" or "I won't get hit by a meteorite" without much risk. But would you feel comfortable predicting the course of AI over the next century? What if you noticed that most other people only managed to win $20 before they slipped up? Remember, if you say even one false statement under such a deal, all of your true statements you've said over years and years of perfect accuracy won't be worth the hole you've dug yourself.
Or – let me give you another intuition pump about how hard this is. Bayesian and frequentist statistics are pretty much the same thing [citation needed] – when I say "50% chance this coin will land heads", that's the same as saying "I expect it to land heads about one out of every two times." By the same token, "There's only a one in a million chance that I'm wrong about this" is the same as "I expect to be wrong on only one of a million statements like this that I make."
What do a million statements look like? Suppose I can fit twenty-five statements onto the page of an average-sized book. I start writing my predictions about scientific and technological progress in the next century. "I predict there will not be superintelligent AI." "I predict there will be no simple geoengineering fix for global warming." "I predict no one will prove P = NP."
War and Peace, one of the longest books ever written, is about 1500 pages. After you write enough of these statements to fill a
War and Peace sized book, you've made 37,500. You would need to write about 27
War and Peace sized books – enough to fill up a good-sized bookshelf – to have a million statements.
So, if you want to be confident to the level of one-in-a-million that there won't be superintelligent AI next century, you need to believe that you can fill up 27
War and Peace sized books with similar predictions about the next hundred years of technological progress – and be wrong – at most – once!
...
A claim like "one in a million chance of X" not only implies that your model is strong enough to spit out those kinds of numbers, but that there's only a one in a million chance you're using the wrong model, or missing something, or screwing up the calculations.
A few years ago, a group of investment bankers came up with a model for predicting the market, and used it to design a trading strategy which they said would meet certain parameters. In fact, they said that there was only a one in 10^135 chance it would fail to meet those parameters during a given year. A human just uttered the probability "1 in 10^135", so you can probably guess what happened. The very next year was the 2007 financial crisis, the model wasn't prepared to deal with the extraordinary fallout, the strategy didn't meet its parameters, and the investment bank got clobbered.
...
[snip further exposition and commentary]
