I spent years watching smart people—PhDs, surgeons, economists, forecasters—defend their predictions with absolute confidence. Then I learned about the calibration experiment, and I realized something unsettling: the smarter they were, the more likely they were to be confidently wrong.

The Experiment That Changed How I Listen to Experts

The calibration test is deceptively simple. I give you a factual question: "What is the distance from Earth to the Moon in miles?" You respond with a range—say, "220,000 to 240,000"—such that you are 98% confident the true value falls inside it.

A perfectly calibrated person would be right 98% of the time. For real humans—including self-styled experts—the true answer falls outside the given range roughly 25% to 30% of the time. That 98% confidence corresponds to actual accuracy of around 70%.

This isn't a small error. It's enormous, systematic, and it holds across decades of research and thousands of subjects. Even after learning about the bias, most people barely improve.

What astonishes me most is that the pattern inverts with expertise. Novices underestimate their confidence slightly; experts overestimate theirs dramatically. The more credentialed someone is, the wider the gap between what they believe they know and what they actually know.

Why Knowledgeable People Are Often Less Calibrated

The mechanism is straightforward: expertise inflates confidence without improving accuracy in the tails.

A radiologist reviewing chest X-rays to detect lung cancer is well-calibrated. The domain has rapid feedback, thousands of cases, and stable underlying rules. The radiologist's subjective confidence and actual accuracy align closely.

A macroeconomist forecasting GDP growth three years out is catastrophically miscalibrated. The domain is Extremistan; the rules change; feedback arrives too late to update; confidence becomes a liability rather than an asset. Yet the economist feels more confident, not less, because their training in models and theory demands certainty.

I've watched this play out across domains. The PhD forecaster performs worse than a naive baseline that says "next year will look like this year." The surgeon systematically underestimates post-operative mortality—studies find their stated "95% survival" patients survive closer to 80% of the time. The security analyst calls the rare event "impossible," then explains confidently why it was "obvious" after it happens.

The pattern is so consistent that it's structural, not accidental. Expertise in a domain tends to inflate confidence relative to accuracy when:

Unfortunately, those conditions describe most of the domains where we desperately need calibrated judgment: macroeconomics, geopolitics, financial markets, long-term technology forecasting, medicine, and public policy. These are also the domains where experts are most confidently wrong.

The PhD Forecaster vs. the Weather App

Let me give you a concrete example. A climate economist with two decades of experience forecasts GDP growth three years out. A graduate student with a naive model—"next year will be 95% as good as this year"—makes the same forecast.

Across many forecasts measured against outcomes, the graduate student wins. The economist's elaborate model adds confidence without adding accuracy. In fact, the economist often performs worse than the naive baseline, especially over longer horizons.

This pattern is not specific to economics. It recurs in political forecasting, clinical prognosis, parole decisions, and weather beyond a few days out. Philip Tetlock's twenty-year study of 284 political forecasters found that the average expert performed barely better than chance and slightly worse than a simple "last year's pattern continues" baseline. The more famous the expert, the worse their calibration.

Why? Because fame goes to confident, bold predictions—which tend to miss. The humble forecaster who says "I don't really know" doesn't get booked on cable news.

The Surgeon's Overconfidence Problem

This is where epistemic arrogance stops being a theoretical problem and starts being a matter of life and death.

Surgeons routinely estimate the post-operative mortality of their patients. Studies tracking these estimates against actual outcomes show a consistent pattern: surgeons underestimate mortality by significant margins. The surgeon who says "95% of my patients survive this procedure" is describing their subjective confidence. The actual figure is closer to 80%.

The surgeon is not lying. The surgeon is not careless. The surgeon is epistemically arrogant in exactly the way every other expert is: their subjective confidence in their own judgment substantially exceeds the judgment's actual accuracy.

Patients making life-or-death decisions deserve to know this number. Almost none are told.

I chose surgery because it's especially important, but the same bias appears everywhere. The financial advisor who is 90% confident in their asset allocation (actual accuracy: 60%). The medical diagnostician who has seen a thousand similar presentations and is certain about the diagnosis (base rates say 70% accuracy). The intelligence analyst briefing the president with the full authority of classified information (later studies find accuracy barely above chance).

The gap between subjective confidence and actual accuracy is largest in high-stakes domains where feedback is delayed, where the test set never repeats perfectly, and where the person's status depends on projecting certainty.

Why Your Intuitions Are Worse Than You Think

Here's what makes this problem recursive: knowing about epistemic arrogance doesn't fix it.

I could tell you right now that you are overconfident in your own estimates. You might nod along, acknowledge it intellectually, and then make a confident prediction about something in your own domain—and be wrong by the same margin as everyone else.

The reason is that epistemic arrogance is not a weakness of the untrained or the unintelligent. It is how trained and untrained minds alike process uncertainty when the stakes are high and the domain is complex.

You cannot reason your way out of it. The confidence feels justified because it's based on genuine expertise. The surgeon has performed a thousand procedures. The economist has studied growth models for decades. The forecaster has called major events before. None of that guarantees accuracy on the next prediction—especially not in Extremistan—but all of it feels like it should.

The Double Trouble of Arrogance: You Don't Know What You Don't Know

Epistemic arrogance has two components that compound:

  1. Overestimation of what you know — You give narrower ranges than warranted. You say "between 2% and 3% growth" when the true distribution is "between 0% and 5%."

  2. Underestimation of what you don't know — You fail to include enough epistemic humility for your own ignorance. You don't know what rules might change. You don't know what categories might emerge that you're not even thinking about.

The combination is devastating. You're simultaneously too confident about what you think you've nailed and too blind about what you haven't even conceived of.

This is why Black Swans are so devastating to overconfident experts. The expert's model works perfectly—until a variable the model didn't include becomes the dominant driver. Then the expert is not just wrong; they're wrong about what could possibly go wrong.

The Practical Fix: Double Your Intervals

After decades of calibration research, the practical takeaway is strikingly simple.

Whatever confidence interval feels right to you on any estimate, double its width.

You'll still be somewhat overconfident, but you'll be far less fatally so.

A forecaster who feels confident saying "growth will be between 2% and 3%" should report "growth will be between 0.5% and 4.5%."

The second range looks embarrassingly wide. It looks uncertain in a way that feels almost admissions of ignorance. That embarrassment is information. The actual world is embarrassingly uncertain, and the narrow range was a polite fiction that nobody wanted to confront.

I've tested this rule on myself repeatedly. My instinct is always to narrow the range until it looks respectable. But the research is clear: my narrow range is usually too narrow. Doubling it doesn't make me accurately calibrated, but it moves me in the direction of not being catastrophically wrong.

Here's another version of the same rule that I find useful: Whatever your forecast is, assume you are 30% wrong about the direction and 50% wrong about the magnitude.

If you say "oil will be $80 per barrel," you're building in the possibility that you've completely misread the direction (it could be $30 or $150) and overestimated your precision about where it lands (it could easily be $120 or $40).

Why This Matters More Than You Think

Epistemic arrogance is not a philosophical problem. It's a practical one that affects decisions with real consequences.

Banks used overconfident risk models and blew up in 2008. Governments made overconfident predictions about how long wars would last. Corporations launched overconfident forecasts about market demand. Medical professionals made overconfident diagnoses. Technology companies made overconfident bets on which platforms would dominate.

Every major catastrophe I can think of involved someone somewhere being more confident than the facts warranted.

The fix is not better expertise. Expertise often makes the problem worse. The fix is structural: build in margin for the errors you can't currently see. Require confidence intervals wider than feel comfortable. Penalize point forecasts that lack error bars. Demand that experts specify in advance what would make them wrong—and do not accept the "I was almost right" defense.

Most importantly: before you trust someone's prediction, ask them to show you their track record against naive baselines. Ask them how often their stated confidence matched their actual accuracy. If they can't or won't answer, treat them as a skilled storyteller, not an accurate forecaster.