Every organization claims innovation to be their most valuable capability, yet nearly every management practice rewards its opposite: more utilization, more KPIs, more dashboards. This again results in less idle time, less ambiguity or variance, and eventually less or no innovation.

This doesn't mean these decisions are irrational, but taken together they raise an uncomfortable possibility.

Perhaps organizations do not lose innovation because competitors become smarter. Perhaps they lose it because success systematically teaches them to stop exploring.

They become prisoners of their previous success, as organizations rarely become less intelligent overnight. Rather, they progressively become less curious...

The central argument of this essay is simple: innovation is not an organizational capability that can be optimized directly. It is an emergent property of preserved exploratory capacity. Organizations lose innovation not because they stop investing in it, but because optimization progressively removes the conditions under which discovery occurs.

Introduction

I've long been fascinated by the history of civilizations and the philosophies they develop to organize work, society, and everyday life. Japan has always held a particular appeal in that regard. Its rich vocabulary for concepts such as continuous improvement, craftsmanship, sustainability, and purposeful living offers one of the most sophisticated philosophical frameworks for optimization ever developed... Whether modern Japan consistently lives by those ideals is, of course, a different question. For years, however, one puzzle remained difficult for me to understand: how could a society so deeply associated with discipline, refinement, and excellence experience decades of economic stagnation? The apparent contradiction only began to make sense after viewing Japan through James March's framework of exploration versus exploitation. That perspective transformed what had seemed like a historical anomaly into a strategic dilemma that extends far beyond Japan itself.

This article is not about Japanese culture. It is about a strategic dilemma that Japan illustrates particularly well.

On a rainy Thursday in February 2024, traders at the Tokyo Stock Exchange applauded as the Nikkei finally surpassed its previous peak from 29 December 1989. For thirty-four years, that number had symbolized a deeper question: What happened to Japan?

The usual explanation begins with the collapse of the late 1980s asset bubble, followed by decades of weak lending, investment, spending, and deflation. However, that explains how Japan slowed, not why. The deeper story reflects the dilemma James March described as exploration versus exploitation.

Exploitation means refining what already works. Exploration means discovering what might work.

Few nations mastered exploitation better than Japan; yet success creates its own habits...

The capabilities that once made Japan exceptionally competitive became less valuable, not because they failed, but because the environment changed. A system built to produce the world's finest cassette or CD player suddenly faced competitors trying to eliminate those products altogether.

The more interesting question, then, is: What happens when the world's greatest optimization engine enters a world where exploration matters more than refinement?

The Paradox

Organizations speak constantly about innovation: in annual reports, strategy documents, boards and executives directives. Entire departments exist to cultivate it. Yet something curious happens inside many successful organizations.

As they become better at execution, they often become worse at discovery. Processes become more efficient. Decisions become increasingly data-driven. Performance becomes easier to measure. Idle time disappears. Meetings become more structured. AI removes friction. Specialists become increasingly (more) specialized. Each decision appears rational; collectively, however, they raise an uncomfortable question: What if organizations do not lose their ability to innovate because markets change, technologies evolve, or competitors move faster? Or, what if they lose it because success gradually teaches them to stop exploring?

This possibility is more significant than innovation alone, as it inherently concerns something more fundamental: how organizations think!

Innovation is not a capability that can simply be installed through workshops, team building events, or incentive schemes. It is often the visible consequence of something less obvious: an organization's ability to continue discovering what it does not yet know.

Every organization possesses an invisible cognitive search space.

I use the term cognitive search space to describe the range of questions an organization remains capable of asking, the anomalies it can still recognize, and the futures it is prepared to explore. Like physical infrastructure or financial capital, this search space can expand or contract over time.

Many organizations celebrate innovation while unintentionally eliminating the conditions that produce it. Most operational dashboards primarily seek utilization, predictability, and measurable output. Yet the ideas that secure an organization's future rarely emerge from fully optimized systems.

Perhaps the real strategic asset is not innovation itself but preserving the conditions under which innovation remains possible..?

Evolution preserved exploration because no organism can survive indefinitely by exploiting yesterday's discoveries.

Optimization Culture

Optimization is one of human civilization's greatest achievements. Globalized logistics networks move goods across continents with remarkable precision. Semiconductor manufacturing, aviation, healthcare and cloud computing, all depend upon extraordinary levels of coordination, standardization, and continuous improvement. Organizations that fail to become more efficient rarely survive long enough to pursue ambitious futures. Efficiency, therefore, is not the enemy. It is one of the primary reasons modern societies function at all.

The challenge arises when optimization quietly expands beyond processes and begins shaping how people think, communicate or how they make sense of the world. This transition is rarely intentional... It develops gradually through a series of - individually highly reasonable, decisions. A dashboard reduces uncertainty by translating complex systems into measurable indicators. A workflow eliminates unnecessary conversations by codifying routine decisions. Reporting structures clarify accountability. Artificial intelligence summarizes lengthy documents into concise insights. Meetings become shorter and more focused. Each improvement appears beneficial in isolation, and often it genuinely is.

Yet every optimization also defines what is no longer necessary. As routines become standardized, opportunities for informal conversations vaporize. As metrics become more comprehensive, attention shifts toward what can be measured, while subtler signals receive less consideration. As decisions become increasingly automated, fewer moments remain in which unexpected observations or unconventional questions naturally emerge.

Highly optimized organizations become exceptionally good at executing what they already understand. Their processes become faster, more reliable, and increasingly predictable. At the same time, they may become progressively less capable of discovering what they do not yet know. Novel ideas often arise from ambiguity, chance encounters, incomplete information, or conversations that appear inefficient in the moment. These are precisely the spaces that optimization tends to compress.

For this reason, every optimization should be understood as more than an efficiency gain. It is also a choice about the kinds of interactions, observations, and possibilities that become less likely. Every improvement expands capability in one dimension while narrowing it in another. The strategic question is therefore not whether organizations should optimize, it is what forms of discovery they are willing to preserve alongside efficiency.

Every optimization is also a decision about what an organization will no longer discover.

Why Evolution Preserved Exploration

The tension between efficiency and discovery isn't really unique to organizations. Long before corporations, markets, or management theories existed, evolution confronted exactly the same challenge. Every intelligent organism must continually balance exploiting what it already knows with exploring what it has yet to learn. A predator that relies only on familiar hunting grounds eventually starves when its environment changes. While others which constantly experiment with new strategies may actually never catch enough prey to even survive. Intelligence emerges from maintaining a dynamic equilibrium between the two.

Perception itself is an active process shaped by expectation, prior knowledge, and continued adaptation. Learning occurs when reality deviates from these expectations in meaningful ways. Cognition is not a collection of isolated functions but a flexible system capable of reorganizing itself in response to new experiences. The brain succeeds because it preserves enough stability to act confidently while remaining sufficiently open to revise its own models of reality.

This capacity depends upon activities that often appear economically unproductive. Children devote countless hours to play without producing anything tangible. Adults drift into daydreams while accomplishing little in measurable terms. People compose music, tell stories, and create rituals, all while consuming time and energy that could be directed toward more immediate goals. Yet these behaviors appear in virtually every known human culture. Evolution rarely preserves costly activities across millennia without conferring significant adaptive advantages.

Their value becomes clearer when viewed through a broader lens. Playing allows experimentation without catastrophic consequences, enabling children like adults to experience possible futures before they become reality. Stories carry knowledge across generations while creating shared mental models that allow strangers to cooperate.

These perspectives can be applied to technology as well, as communication systems and technologies are never merely technical infrastructures. They embody assumptions about how knowledge should circulate, who participates, and which forms of interaction are considered valuable. Every communication architecture amplifies certain possibilities while constraining others. Therefore, technology also transforms expertise from something embodied in individuals into systems that distribute and automate knowledge. These developments create remarkable efficiencies, but they also alter where judgment, creativity, and unexpected insight can emerge.

Historian John Darwin offers a comparable perspective on a much larger scale. His work on empires emphasizes that long-term success rarely resulted from perfect planning or centralized optimization alone. Complex societies endured because they adapted continuously to changing environments, negotiated local conditions, and developed resilience by absorbing (unexpected) developments. Flexibility often proved more durable than rigid efficiency.

Taken together, these perspectives suggest that many behaviors appearing inefficient in the short term perform indispensable adaptive functions over longer timescales. Exploration, reflection, play, and creative expression generate resilience by expanding the range of future possibilities before they become necessary. Evolution appears to have recognized this trade-off long before humans attempted to formalize it in organizations.

If biological intelligence evolved mechanisms to preserve exploration, can human institutions do the same? History suggests the answer is surprisingly inconsistent.

Intelligence depends as much on preserving uncertainty as on reducing it.

Optimization in Organization

During the Cold War, Soviet scientists envisioned ambitious national computer networks capable of transforming economic planning. Among the most notable proposals was OGAS, a nationwide system intended to connect factories, ministries, and planning agencies into a unified information network. Technically, many of its ideas anticipated concepts that would later become central to distributed computing and digital information systems. Its failure is often attributed simply to technological limitations or insufficient computing power. Yet the deeper challenge lay elsewhere.

Large planning systems excel at organizing information that is already known. They aggregate data, coordinate resources, and optimize decisions within existing models. Innovation, however, depends upon discovering information that no central planner, algorithm, or institution yet possesses. New knowledge rarely appears fully formed. It emerges through countless local experiments, disagreements, unexpected observations, failed attempts, and anomalies that initially resist explanation. The central problem was therefore never computers alone. It was organizational architecture.

How should a society organize learning under conditions of uncertainty?

For me, organizational cognition consists of four capacities:

• Observe,
• Interpret,
• Execute,
• Explore.

Optimization strengthens interpretation, automation strengthens execution, exploration preserves observation. Lose one and the system becomes fragile.

Industrial management primarily sought to optimize physical labor. Factory managers measured machine utilization, standardized production processes, and minimized idle hands. Today's organizations increasingly optimize cognitive labor instead. They measure attention, compress communication, automate judgment, and accelerate decision-making. The object of optimization has shifted from muscles to minds.

This distinction matters because cognitive work differs fundamentally from mechanical production. Reflection often appears unproductive because its most valuable outcomes cannot be predicted in advance. Curiosity rarely aligns neatly with key performance indicators. Unforced conversations resist scheduling. Insight frequently emerges only after periods that, from the perspective of efficiency metrics, appear unstructured or even wasteful. As Bertrand Russell observed decades ago, many of civilization's greatest advances have depended upon time that was free from immediate productive demands. The conditions that generate understanding are often invisible to the systems designed to measure output.

The consequences of over-optimizing cognitive work will not appear as immediate failure. Organizations continue executing familiar tasks with increasing precision and efficiency. Existing products improve. Processes become smoother. Decisions become faster. Yet beneath these visible gains, the space available for exploration gradually shrinks. Questions that cannot justify themselves through measured metrics will receive less attention while unusual ideas become harder to pursue. Information that does not fit established categories is increasingly filtered away before anyone has the opportunity to investigate it.

Eventually, the external environment changes... New competitors emerge from unexpected directions. Technologies converge. Customer expectations shift. Assumptions that once seemed permanent gradually dissolve. At this point, organizations often attribute great strategic surprise to forces outside their control. However, that surprise reflects something more subtle...

Over years of successful optimization, they may have gradually abandoned precisely those capabilities that once enabled them to notice weak signals, question prevailing assumptions, and explore unfamiliar possibilities.

The lesson extends well beyond Cold War planning or the current artificial intelligence wave. Every organization must optimize in order to survive. The more difficult question is what forms of exploration it chooses to preserve while doing so. Long-term resilience depends not only on executing existing knowledge more efficiently, but on maintaining the institutional capacity to discover knowledge that no optimization process can specify in advance.

No system can centrally optimize knowledge that has not yet been discovered.

What about Innovation Management?

If these observations hold, then innovation itself may be a misleading objective. The governance question therefore changes fundamentally. Instead of asking how an organization can become more innovative, leaders might ask a different set of questions.

Which forms of exploration are gradually disappearing from everyday work, and how can we track them?
Which cognitive capabilities should remain embedded within people rather than delegated entirely to systems?
Which activities appear inefficient today but function as strategic investments in future adaptability?
At what point does additional optimization begin to reduce, rather than expand, an organization's ability to learn?

In summary: What evidence do we have that our cognitive search space has not contracted?

These are not primarily operational questions. They are questions of governance and boards already routinely oversee financial capital, technological investment, operational resilience, regulatory compliance, and enterprise risk. Increasingly, they may also need to oversee something less tangible but equally consequential. The organization's exploratory capacity, like:

• diversity of experiments,
• proportion of resources devoted to exploration,
• horizon scanning,
• number of assumptions explicitly challenged,
• optionality in capital allocation,
• etc.

This is not because every experiment succeeds, nor because every unconventional idea deserves investment. It is because organizations that gradually lose the ability to generate surprises eventually become vulnerable to surprises generated elsewhere.

It's therefore crucial to distinguish between exploitation and exploration, as healthy institutions require both. Exploitation refines existing capabilities, improves efficiency, and delivers reliable performance. Exploration searches for alternatives, accepts uncertainty, and generates the knowledge upon which future performance depends. The difficulty is that exploitation produces visible and immediate returns, while exploration often appears costly, ambiguous, and difficult to evaluate until long after investments have been made. Good governance therefore requires protecting activities whose value cannot yet be demonstrated with certainty.

Systems theorist Stafford Beer approached the problem from another direction. He described organizations as living systems whose long-term viability depends upon maintaining the capacity to adapt as environments change. Governance, in this view, is not simply the optimization of current operations. It is the continuous balancing of stability and adaptation. We may also argue that organizations should not merely become more efficient at doing things right, but should regularly question whether they are doing the right things at all. Learning, therefore, cannot be reduced to the accumulation of data. It depends upon preserving the ability to rethink the assumptions that generated the data in the first place.

Elinor Ostrom's work on the governance of complex systems offers a complementary lesson. Institutions that endure over long periods rarely depend upon rigid central control or unrestricted decentralization alone. Instead, they cultivate rules that enable experimentation, feedback, local knowledge, and adaptation across multiple levels simultaneously. Their resilience derives less from perfect optimization than from preserving the capacity to respond when conditions change - in one way or the other.

This does not imply that organizations should reject efficiency, nor does it suggest returning to vague notions of creativity for its own sake. Optimization remains indispensable. Without disciplined execution, organizations cannot scale, compete, or survive. The question is not whether optimization should occur, but what it should optimize, and what it should intentionally leave unoptimized.

Optimization and exploration are therefore not competing philosophies. They are complementary functions operating across different timescales. One preserves present performance by refining existing knowledge. The other preserves future adaptability by creating opportunities to discover knowledge that does not yet exist.

Neither can substitute for the other. Organizations that neglect optimization eventually lose competitiveness. Organizations that neglect exploration eventually lose relevance.

This also applies to cyber... Most security teams optimize for exploitation, while attackers remain exploration-heavy - no pun intended.

The central strategic challenge is therefore one of governance. It is the design of institutions capable of balancing execution and discovery without allowing either to dominate completely. Biological intelligence appears to have converged on this balance through millions of years of evolution, continuously refining known behavior while remaining sensitive to unexpected change. Human institutions are still learning how to achieve the same equilibrium. Perhaps the organizations that prove most resilient in the decades ahead will not be those that optimize the fastest, but those that govern most wisely the delicate boundary between efficiency and discovery.

Every optimization expands present capability while narrowing future possibility.

Summary and Outlook

Thinking is not the production of answers, it is the continual revision of internal models.

Perhaps this explains why so many organizations struggle to remain innovative despite investing ever greater resources in innovation itself. New departments are established, innovation laboratories are created, accelerators are funded, and increasingly sophisticated technologies are deployed in the hope that discovery can be engineered as reliably as production. Yet these initiatives often focus on manufacturing visible outcomes while leaving largely unchanged the organizational conditions from which those outcomes naturally emerge.

Curiosity cannot simply be outsourced, even to highly capable external partners or increasingly powerful artificial intelligence. Judgment cannot be delegated indefinitely without altering the capabilities of those who once exercised it. Exploration itself cannot become fully optimized without gradually narrowing the very search space upon which meaningful discovery depends. Every decision that reduces uncertainty simultaneously defines which uncertainties will no longer be explored. In many situations, that trade-off is entirely appropriate. Organizations require stability, coordination, and reliable execution. Yet there are moments when the cumulative effect of thousands of individually sensible, small optimizations quietly begins to constrain future adaptability.

Recognizing where that boundary lies may become one of the defining governance challenges of the coming decade. As organizations accumulate unprecedented quantities of data, automate increasingly sophisticated forms of reasoning, and optimize ever larger portions of cognitive work, the temptation will be to regard uncertainty itself as a problem to be eliminated. Yet uncertainty is also the environment within which learning occurs.

Professional practice not only depends upon technical expertise but upon reflection in situations difficult to address through routine solutions. Likewise, organizations capable of sustained success are distinguished less by their efficiency than by their ability to learn continuously as conditions evolve. Much of what individuals and organizations know cannot be fully articulated or codified. Tacit knowledge develops through experience, participation, and attentive engagement with the world.

It is this form of knowing that becomes difficult to preserve when every activity is redesigned primarily for standardization, automation, and measurement. The challenge is not simply that machines may perform certain tasks more efficiently. It is that organizations may gradually lose the human capacities through which entirely new questions are first discovered.

Strategic surprise frequently appears sudden only because the capacities required to perceive it disappeared quietly over many years.

Perhaps this is the deeper lesson. Innovation is not the capability organizations should seek to maximize. It is not an operational target, a quarterly metric, or a process that can be reliably commanded into existence. Rather, innovation is evidence that something more fundamental remains intact. It signals that an organization has preserved the cognitive, social, and institutional conditions under which genuine discovery is still possible.

The organizations that endure may not be those that optimize the fastest. They may be those that preserve the widest cognitive search space.

Boards already ask whether organizations are executing efficiently... Perhaps the more important question is whether they are still capable of discovering effectively. Organizations do not become adaptive because they possess better answers, they become adaptive because they continue asking better questions.

Further Reading

• K. Weick, Sensemaking in Organizations
• M. Polanyi, The Tacit Dimension
• J. March, Exploration and Exploitation in Organizational Learning
• A. Clark, Supersizing the Mind
• D. Levinthal's work on organizational search