How Innovation Actually Works: Trial and Error vs Planning

We tell stories about innovation that are clean, linear, and wrong.

The story goes: a genius has an insight, develops a theory, tests it, creates a product, brings it to market. Insight → theory → product → success.

The reality is messier. Innovation usually works like this: someone tries something, it fails. They try something else based on what failed. They stumble on something that works. Someone else sees what worked and refines it. Years later, someone writes the theory explaining why it works.

Theory comes last, not first.

Table of Contents

The Arrow of Knowledge

Knowledge does not flow from theory to practice. It flows from practice to theory.

We observe that our muscles get stronger when we lift heavy things. Later, we develop theories about muscle physiology and protein synthesis. The practice (lifting) came first. The theory came later.

We observe that penicillin kills bacteria when a contaminated petri dish is left on the bench. Later, we develop theories about antibiotics. The practice (Fleming's accident) came first. The theory came later.

We observe that Javascript makes web pages interactive. We develop theories about how programming languages should work. The practice came first.

The arrow of knowledge is from practice to theory. Not theory to practice.

Yet the way we teach, the way we fund research, the way we structure education, all assumes the opposite: develop theory, then apply it.

This is the Taleb insight: innovation is predominantly achieved through tinkering, trial-and-error, and accidental discovery — not through directed research and theoretical planning.

Trial and Error in History

Evidence everywhere:

The wheel on luggage: Six thousand years after the wheel was invented, someone finally attached wheels to luggage. We sent humans to the moon before we did this. Why? Because innovation isn't the result of intelligence applied to obvious problems. It's the result of someone getting frustrated and tinkering.

Penicillin: Alexander Fleming wasn't researching antibiotics when he discovered penicillin. He was researching something else. A contaminated petri dish (a mistake) showed him that mold killed bacteria. The discovery was an accident. The theory came later.

The jet engine: Developed by tinkering engineers, not by theoretical physicists. The engineers didn't have complete understanding of thermodynamics and fluid mechanics. They built, tested, failed, and iterated. Theory couldn't have predicted the design.

The Industrial Revolution: Driven by tinkerers and hobbyists, not by scientists. The steam engine, the textile machinery, the manufacturing processes — all developed through trial and error before the theoretical understanding was in place.

Aspirin: Developed through trial and error. Later, we understood the mechanism (inhibition of prostaglandin synthesis). But the drug worked before anyone understood why.

The Personal Computer: Built by hobbyists in garages, not by IBM or Xerox (who had the resources and the theory). The garage tinkerers had no grand plan. They were solving specific problems for themselves.

The Green Lumber Fallacy

Joe Siegel was a spectacularly successful trader in green lumber.

For years, he thought "green lumber" meant lumber painted green. It actually means freshly cut, undried lumber.

He was wrong about the most basic fact of the commodity he traded. Yet he consistently outperformed traders with sophisticated knowledge of lumber markets, supply chains, and economic theory.

Why? Because what actually predicted lumber price movements — inventory dynamics, order flow, dealer positioning — had nothing to do with what he thought he knew.

The theoretically knowledgeable traders had the wrong information in their heads. Joe had the right information about what actually moved prices, even though he was wrong about the definition of the thing itself.

This is the Green Lumber Fallacy: what you think matters and what actually matters are often completely different.

Applied to innovation: what theorists think will drive innovation (deep understanding, careful planning) is often not what actually drives it (accident, frustration, trial and error).

The person who tinkered with something and made it work might have zero theoretical understanding of why it works. But they know what actually works. The theorist with perfect understanding of the theory might miss what actually matters.

The Lecturing Birds How to Fly Effect

Observers see birds flying and ornithologists teaching about bird flight. They conclude that the teaching caused the flying.

This is an epiphenomenon — the teaching and flying co-exist but one didn't cause the other.

Applied to economics: universities exist and the economy grows. So universities cause economic growth, right? The logic is backwards.

Economic prosperity causes investment in education. Not the reverse.

Applied to innovation: great research exists and technological advancement happens. So research causes advancement? Often backwards. Economic development causes investment in research to explain what already works.

Most major technological advances have emerged from practice — someone building something, tinkering, experimenting. Later, universities systematize the knowledge and teach it.

The mistake is thinking the teaching caused the innovation. The reality is: the innovation happened, then was explained by theory, then was systematized in universities, then was taught.

The Rational Flâneur vs. Planning

The alternative to directed planning is the rational flâneur.

A flâneur is a wanderer, someone who drifts through the city without a fixed itinerary, alert to whatever is interesting.

A rational flâneur is someone who makes decisions opportunistically, adapting as new information arrives, without a fixed destination.

Compare two ways to innovate:

The planner: Develop a five-year strategy. Identify the market opportunity. Allocate resources. Execute the plan. Measure against goals.

The flâneur: Start with a problem you want to solve. Build a solution. See what works. Adapt based on what the market wants. Follow opportunities as they emerge.

History suggests the flâneur approach produces more innovation. Steve Jobs famously didn't do market research. He built what he believed people should want. Howard Schultz visited Italy, noticed the coffee bar culture, and adapted it to the US. Neither had a detailed plan.

The planning approach feels more scientific and professional. The flâneur approach feels ad-hoc and unprofessional. But the empirical evidence favors the flâneur.

The Teleological Fallacy

Teleology is the belief that intelligent action is directed toward a known end goal.

The teleological fallacy is the assumption that all successful innovations came from clear intention. The person knew what they were building. The plan worked.

Actually: the plan rarely works. The path is rarely straight. Success is rarely the result of clear forward vision.

But after success arrives, we write the history backward. We tell the story as if it was planned all along. This narrative bias makes us think planning is more effective than it is.

The iPhone example: Apple didn't do market research asking "do you want a touchscreen phone without a physical keyboard?" Consumers couldn't articulate wanting something that didn't exist. The market didn't "demand" the iPhone. Apple built it anyway, based on Steve Jobs' intuition.

The post-hoc narrative: "Apple identified the smartphone market opportunity and dominated it with superior innovation." The actual history: "Steve Jobs believed in something, built it, and the market eventually agreed."

The teleological fallacy makes us credit planning when success was actually the result of optionality, trial and error, and luck.

Why We Tell Clean Stories

We prefer clean stories because they make us feel like we understand and control outcomes.

"Innovation comes from careful planning and execution" is a story we can teach, measure, and replicate. It's a story that makes sense in a business school context.

"Innovation comes from tinkering, accident, and stumbling on something that works" is a story that's harder to teach. It's messier. It suggests less control.

So we teach the clean story. We fund based on the clean story. We hire based on the clean story. We write histories using the clean story.

But the messy reality is that tinkering, trial and error, and accident produce most actual innovation.

This creates a systematic bias: we're incentivizing the wrong process while celebrating the outcomes of the right process.

Common Misreadings

Misreading 1: Innovation is random so planning is useless.

No. Trial and error is not random. It's structured iteration: try something, learn from failure, try something else based on what you learned. That's more intelligent than planning to specific objectives that turn out to be wrong.

Misreading 2: Only geniuses can innovate this way.

False. Genius helps, but the structure of trial-and-error is available to anyone. Fleming didn't need to be brilliant to notice that the contaminated plate was interesting. Joe didn't need to be brilliant to pay attention to what actually moved prices. You don't need genius to tinkering and observe.

Misreading 3: We should abandon planning entirely.

Wrong. Some planning is useful. But planning should be light — establishing a direction, not a specific destination. Then iterate, learn, adapt.

Current Context: Innovation in 2026

(Verify current AI, startup, and tech landscape before publishing.)

Right now, there's immense focus on AI as a transformative technology.

There's also enormous planning around AI: what problems will it solve, which industries will it disrupt, how should companies position themselves.

The planning is probably wrong. The problems AI actually solves will be surprised. The industries actually disrupted will be different from the predictions.

But the innovators who build AI applications, tinkering with what works, will discover the real applications faster than planners trying to execute a perfect strategy.

The innovation will come from: people playing with AI tools, noticing what's surprisingly good, building on those surprises, iterating, and slowly discovering what actually works.

The people who planned carefully to be "ready for AI disruption" will be less prepared than people who tinkered and learned what AI actually does.

This is antifragile innovation: you're structured to benefit from whatever actually emerges, not betting on a specific prediction.