Most of us don't have time to run a twenty-year study on an expert's track record. We need a faster way to tell the real thing from a credential-wearing storyteller.

Here is the checklist I use.

Question 1: Does the domain allow expertise to exist?

Some domains have the conditions necessary for expertise. Others don't.

Domains where expertise exists: Chess, accounting, aviation, dentistry, physics, medicine (narrow domains like oncology, not prognostication), skilled trades.

These domains have tight feedback loops, stable rules, sufficient complexity to reward training, and sufficient volume of cases to learn from.

Domains where "expertise" is often an illusion: Stockbroking, macroeconomics, political forecasting, management consulting, intelligence analysis, long-term technology forecasting.

These domains have broken feedback loops, unstable rules, Extremistan dynamics, and incentives to claim credit for successes while disclaiming failures.

If your expert operates in a domain where expertise doesn't actually exist, discount their confidence by a large factor.

Question 2: What's their track record against naive baselines?

Ask the expert directly: "What is your accuracy against a naive baseline?"

Naive baseline examples: - "Last year's number continues" - "Random walk" (no direction, just random variation) - Simple trend extrapolation ("if growth was 2% last five years, it will be 2% next year") - Index funds (for investment advisors)

If the expert has not been measured against a naive baseline, that's suspicious.

If the expert beats the naive baseline consistently and significantly, they might be worth listening to.

If the expert's track record is indistinguishable from the naive baseline, or worse, the expert has no real advantage. You're paying for entertainment, not forecasting.

Question 3: Do they specify conditions under which they'd be wrong?

This is the falsifiability test.

Ask the expert: "What would have to be true for your forecast to be completely wrong? What would you see that would make you change your mind?"

If the expert cannot name disconfirming evidence, their position is unfalsifiable. It is not knowledge; it is narrative.

"I predicted the stock market would rise, and I was right even though it fell, because I was early" is unfalsifiable. There is no outcome that would falsify it.

A good expert can say: "If X happens, I'm wrong. If we see Y in the data, I'll revise. If Z occurs, the entire framework changes."

Question 4: Do they have skin in the game?

Does the expert bear the downside of being wrong?

If your financial advisor loses money when their advice is wrong, they have skin in the game.

If your financial advisor gets paid the same whether they beat the market or underperform, they don't.

Taleb calls this the "skin in the game" principle: if someone's advice could harm them personally when wrong, you can trust their confidence more.

An expert with skin in the game has a much lower rate of false confidence. They cannot afford the "I was almost right" defense if they're losing their own money.

Question 5: Are they the same people who were wrong last time?

In domains with broken feedback loops, failed forecasters don't disappear. They reposition.

The economist who called the major recession a "soft landing" a month before it started? Still consulting. The intelligence analyst who missed the 9/11 signals by a large margin? Still briefing presidents. The management consultant whose recommendations tanked the company? Moved to a new firm.

There is very little penalty for being wrong in Extremistan domains.

If your expert was wrong about something consequential and was not held accountable, that's information. It suggests their track record is not as clean as presented.

The Questions Combined

Here's how these questions work together:

An oncologist passes all five tests. Trust them more than average.

A political consultant operates in a domain where expertise is questionable.

Their track record is rarely measured against baselines, and when it is, it's mediocre.

They cannot specify falsifying conditions: their forecasts are usually vague enough to accommodate almost any outcome.

They have no skin in the game: they get paid whether right or wrong.

If they were wrong, they were not held accountable.

A political consultant fails most of these tests. Listen to them skeptically, if at all.

Why These Tests Matter

Most people substitute credentials for competence. They see "PhD economist" and assume expertise. They see "investment advisor" and assume the ability to beat the market.

These tests let you go beyond credentials to actual evidence of skill.

Will these tests reject some real experts? Possibly. An expert in a domain with legitimate expertise might not have been measured against baselines before you ask, or might not be comfortable specifying falsifying conditions.

But these tests will reliably reject false experts and empty suits with confidence—which is more important than being generous to every credentialed person.

The best expert is the one who:

Few experts meet all five criteria. Those who do are worth listening to.