Base Rate Neglect: Why Your Brain Ignores Statistics and Trusts Stories

You’re scoping a new product build, and you’ve done the responsible thing. Before you let yourself fall in love with the plan, you went and found the base rate. Projects like this one tend to run about 40% over their first estimate. Good. You wrote it down. Now you sit there with that number and you do exactly what almost everyone does next: you decide it doesn’t apply to you. Those overruns came from teams that didn’t scope as carefully as you have, didn’t have your engineers, didn’t see the risks coming. You nod at the 40% and plan for 10 anyway.

That’s the part the advice to “check the base rate” never warns you about. Finding the number is the easy half. The hard half is that the moment you have it, your own situation starts arguing with it, and the argument almost always wins. Knowing base rates exist doesn’t protect you. The work is in which rate you pick, how you read it, and what you do when every instinct tells you you’re the exception.

The evidence

Start with how completely a story beats a statistic, because the effect is sharper than you’d guess. This is base rate neglect, among the most reliably reproduced findings in decision research. Give people a group that’s 70% engineers and 30% lawyers, then hand them a personality sketch that sounds vaguely lawyerly, and they’ll call the person a lawyer at 80 or 90% odds. The actual makeup of the group, which is the single most useful fact they have, gets treated as if it weren’t there. A description does it, and the description need not even be true, only vivid.

Now watch the same failure cost real money. When teams plan large projects, overruns turn out not to be the occasional bad break. They’re the standing pattern. Rail projects run roughly 45% over budget on average. Big technology projects run around 27% over. This holds across decades and across countries, which rules out the comfortable explanation that those were just badly run jobs. Every one of those teams had a detailed plan they believed in, and nearly every one of them planned as if the pattern were about other people.

The cleanest demonstration of why is also the smallest. Ask people to predict when they’ll finish their own work and they build the estimate from the specifics: this plan, these steps, this week. In one well-known study the average prediction came in around 34 days and the real average was closer to 55. The same people, asked instead how long past jobs like this had actually taken them, gave far better estimates. The accurate answer was sitting in their own history the whole time. They just didn’t reach for it when the question was about themselves.

How it works

Here’s the part that matters and that the base-rate slogan skips. The trouble isn’t that you can’t find the number. It’s that you build your forecast two different ways and only trust one of them.

When you reason from the details of your own situation, your plan, your team, your market, you’re taking what’s called the inside view. It feels rich and specific, so it feels reliable. The catch is that it can only see the path to the outcome you’re picturing. It’s blind to the dozens of ways those same ingredients have failed to produce that outcome before, because those failures aren’t in front of you. They happened to other people, off-screen.

The outside view asks a flatter question: what usually happens in cases like this one? It throws away the texture of your specific story and treats your project as one more draw from a pile of similar ones. It feels generic, almost insulting to the care you’ve put in. It’s also far more accurate, because the pile already contains every clever team that thought their specifics would save them, and most of them were wrong in ways they couldn’t see in advance.

So the failure isn’t ignorance of statistics. It’s that your specifics feel like evidence you’re different, when in truth your specifics are exactly what everyone in the base rate also had.

Your particular plan isn’t evidence you’ll beat the average. It’s the same thing everyone who landed inside the average was also looking at.

That reframe is the whole game. Once you see that “but my situation is different” is the default setting and not a finding, the base rate stops being a number you acknowledge and override. It becomes the thing you have to be argued away from, with real evidence, not enthusiasm.

How to use it

The standard move, writing down what usually happens in cases like this before you weigh the specifics, is the right first step. Two refinements turn it from a slogan into a forecast you can trust.

First, pick the comparison group honestly, because that’s where the manipulation hides. The base rate you get depends entirely on what you decide counts as “cases like this,” and a motivated mind will draw the circle tight enough to flatter the answer. You’re sizing a six-month platform migration and you tell yourself the relevant rate is “projects our own top engineers have run,” which sounds strong, when the rate that actually predicts your outcome is “migrations of this scope across the industry, including the ones that slipped a year,” which is far less rosy. Before you trust a base rate, check whether you chose that reference class because it fits the situation or because it gives you the number you wanted. The honest group is usually broader and less encouraging than the first one you reach for.

Second, read the whole spread, not just the average. A base rate is a distribution, and the average is the least useful thing in it. If similar projects run 40% over on average, the real question is the range: how often do they run double, and what separated the ones that came in near plan from the ones that blew up? Plan against where you’ll actually land across that range, not against the single midpoint, because the midpoint is the one outcome you can be fairly sure you won’t hit exactly.

Then the hard case, the one you’ll meet most: you genuinely believe you’re the exception. Sometimes you are. The test is simple and it stings. Name the specific, checkable reasons your situation will beat the base rate, and ask whether they’d survive someone else reading them cold. “We’re more committed” and “we have a great team” fail, because every team in the base rate said the same. “We’ve shipped two builds of this exact size in the last year, both within 15% of estimate, with this same crew” passes, because it’s a fact a skeptic could verify and most of the failed cases couldn’t claim. The base rate is the benchmark your optimism has to clear, in evidence, before you’re allowed to adjust off it.

Why it matters

The forecasts that hurt you most aren’t the ones where you never looked at the odds. They’re the ones where you looked, felt informed, and then let the vividness of your own plan walk you straight back to the optimistic number you started with. You did the check. The check just didn’t survive contact with how much you wanted to be different.

That’s why this is less about knowing base rates exist, which you already do, and more about a kind of discipline you have to keep applying against your own pull. Pick the comparison group you’d pick if it were someone else’s project. Read the spread, not the comforting midpoint. Make your case for being the exception out loud, in facts a stranger could check, and notice how often it collapses into “but it’s mine.” Most of the time you’ll still go ahead. You’ll just go ahead knowing the real size of the bet, instead of the size your story sold you.