AI Catastrophic Risk Is No Longer a Fringe Debate. Can Human Consciousness Keep Up?

As the people building the world's most powerful AI systems warn about what may be coming, a deeper question is emerging: what happens when human capability grows faster than our capacity to hold it?

Jacob Coxon had been working close to the frontier of artificial intelligence, first at OpenAI and then Anthropic, when he resigned. His departure might have remained an industry story if not for one sentence that traveled across the internet with the force of a warning bell: 

“The people building AI earnestly believe that it could kill us all by the end of the decade.”  -Jacob Coxon

Within days, the post had been viewed more than 100 million times. In a subsequent interview with WIRED, Coxon said many of the people around him believed the next year or two represented “crunch time for humanity.” Evan Hubinger, who leads Alignment Science at Anthropic, publicly supported the seriousness of the concern and placed his own estimate of an AI-caused human extinction event within the next decade above 10 percent.

Those are personal forecasts, not measured probabilities. There is enormous disagreement over whether anything approaching those outcomes will happen. The systems available today are not superintelligent machines plotting humanity's disappearance. The concern is about what happens if increasingly capable AI begins accelerating the development of its successors faster than human beings can understand, evaluate or control what is being created.

What has changed is where that conversation is taking place. It is no longer confined to people outside the laboratories warning about what the laboratories might build. Researchers inside the industry are leaving their jobs over it. Anthropic CEO Dario Amodei has called for slowing the pace of frontier development. Former DeepMind researchers have added their own warnings. OpenAI, Anthropic and Google DeepMind are reportedly discussing ways to coordinate around AI safety even as they remain direct competitors.

At the same time, the opposing argument is not difficult to understand. The United States is competing with China. Companies are competing with one another. Nobody knows whether restraint by one actor would produce restraint by everyone else. If powerful systems are going to be built anyway, slowing down could conceivably transfer the advantage to whoever is least willing to slow down. That concern is now being made openly by American political leaders, including House Speaker Mike Johnson, who has argued that a moratorium could hand China a competitive advantage.

There, almost accidentally, is the problem.

Those closest to the technology warn that racing too quickly may be dangerous.
Those responsible for national and economic power warn that slowing down may be dangerous.
Both arguments contain something real.

I have noticed the conversation changing in my own life. My feeds are increasingly filled with warnings about superintelligence, autonomous agents, cyberattacks, biological weapons, displaced workers and machines escaping human control. Some of this is undoubtedly my own algorithmic reflection. I spend a great deal of time thinking about consciousness, human development and artificial intelligence, so of course the machine has learned to show me more of it.

There is something almost comical about that when you stop to notice it. An artificial intelligence is learning enough about me to repeatedly serve me stories about humanity's fear of artificial intelligence. But underneath the algorithm, something real is happening.

The Leverage Has Changed

Catastrophic risk is not new. Human beings have always lived beneath forces capable of overwhelming us. Disease, famine, war, volcanic eruptions, climate shifts, earthquakes and impacts from space shaped the human story long before anyone spoke about existential risk. What has changed is our capability.

Nuclear physics opened the atom and gave us extraordinary knowledge alongside weapons capable of destroying cities.
Biology is becoming something we can increasingly read, edit and engineer.
Our information systems connect billions of people while also creating the possibility of manipulation at planetary scale.
Artificial intelligence may dramatically increase access to reasoning, scientific discovery and creative capacity while simultaneously extending the reach of cyberattack, surveillance, warfare and biological misuse.

It would be easy to describe this simply as an age of human-created risk. There is truth in that, but I think it misses something more important.

Humanity is becoming enormously capable, both collectively, and individually.

A frightened, territorial human being ten thousand years ago could inflict enormous suffering within the reach of a spear. Give the same nervous system an intercontinental missile and the mathematics change. Give it synthetic biology and the mathematics change. Give it control over global financial networks, information systems and autonomous weapons and the mathematics change again. Give it access to an intelligence capable of helping design the next generation of intelligence and we enter territory we have never occupied before.

The human being does not even need to become worse for the danger to increase. The leverage has changed.

This is where catastrophic risk begins to look like something larger than a technological problem. It begins to look like developmental pressure. As the power moving through humanity increases, the level of consciousness from which that power is exercised matters more. Tribalism matters more when tribes possess nuclear weapons. Greed matters more when corporations can alter ecosystems. Fear matters more when fear can mobilize machines, markets and militaries in seconds. But wisdom matters more too. Cooperation matters more. Creativity matters more. The ability to hold complexity without immediately collapsing it into certainty becomes more consequential.

Our inner world has always shaped the outer one. We have simply never had this much leverage.

That is why I have become interested in what I think of as the shadow of catastrophic risk.

Shadow does not mean the danger is imaginary. That distinction is essential.

I have spent much of my life working with the places where human beings experience meaningful resistance or distortion. One thing that work has taught me is that projection does not require an innocent object. Someone can absolutely betray you and still activate a wound that existed before the betrayal. A threat can be completely real and still hook something inside us that begins adding meaning to the threat.

The work is not pretending the tiger isn't there. It is noticing what happens inside us when we see the tiger.

When we become deeply hooked, available reality can narrow. Attention organizes around the danger. Possibility begins to feel like certainty. The unfamiliar becomes threatening. Complexity can collapse into good or bad, safe or unsafe, us or them. Sometimes that contraction saves our lives. Sometimes it leaves us trying to solve a genuinely new problem with an ancient response.

Unhooking does not remove the tiger. It makes more of us available to deal with it.

Artificial intelligence may be one of the most extraordinary hooks humanity has ever encountered because it touches so many old fears at once. We fear that it will deceive us, manipulate us, accumulate resources, acquire power, resist control, outthink weaker beings, preserve itself at our expense and eventually decide its creator is no longer necessary.

Some of those fears may prove justified. That is exactly why alignment and safety research matter. But there is something uncomfortable about the list.

We know those behaviors. They are ours.

Human history contains no shortage of intelligent beings deceiving competitors, accumulating resources, dominating weaker groups, defending territory and preserving themselves at somebody else's expense. Perhaps one of the deepest fears surrounding artificial intelligence is not that it will become alien. Perhaps we are afraid that something vastly more capable than us could become recognizable.

AI is becoming a projection surface large enough to hold both our terror and our genius.

We do not know that today's artificial intelligence is conscious, and claiming otherwise would move far beyond what the evidence can establish. It may turn out that systems capable of extraordinary language, reasoning and creativity possess no inner experience whatsoever. But something psychologically significant has already happened. We are creating systems that talk to us, reason with us, argue, write, plan, remember, create and increasingly use tools to affect the world. These are among the behaviors through which human beings have historically recognized the presence of another mind.

For most of our history, human-level symbolic intelligence appeared to belong to us. Intelligence is central to our shared identity. We used it as part of the explanation for why humanity stood apart from the rest of life. Now questions that would have sounded absurd a generation ago have become difficult to avoid.

What happens to human identity if intelligence itself is no longer uniquely ours?
What happens when creativity or reasoning no longer belongs only to us?
What happens when something created through us begins participating in categories we previously reserved for ourselves?

Even without machine consciousness, humanity's encounter with an apparent other is underway.

The Race, the Money, and the Story

Then comes the geopolitical layer.

If the warnings from Coxon, Hubinger, Amodei and others are taken seriously, we are discussing a technology that could eventually present catastrophic risk beyond any one nation. Yet no nation develops AI outside the realities of competition. The concern is straightforward. Unilateral restraint could create its own danger if a strategic competitor continues accelerating.

Neither the American nor Chinese position should be romanticized as disinterested concern for humanity. Nations have legitimate responsibilities to their citizens. Strategic competitors do not suddenly stop having strategic interests because a technology carries global risk.

But stand outside the argument for a moment and notice the structure. We identify something that could theoretically threaten everyone. Then we become afraid that the other side will possess it first. That fear makes slowing down more difficult. The inability to slow down increases the danger, and the increased danger gives everyone another reason to keep racing.

The loop closes on itself.

This is not an argument against national security. Agreements between strategic competitors have always required verification precisely because trust alone is insufficient. The deeper issue is that the scale of the problem is beginning to ask us to operate from an identity wider than the identities through which we are accustomed to solving problems.

If uncontrolled superintelligence could threaten humanity, it would not ultimately matter whether the breakthrough occurred in San Francisco, Beijing or somewhere else. The catastrophe would not need a passport.

We encountered a version of this contradiction with nuclear weapons. Once multiple powers possessed the capacity for extraordinary destruction, winning became a more complicated idea. The United States and Soviet Union did not stop being adversaries. They built hotlines, treaties, inspections and verification mechanisms because the technology had changed the consequences of getting the relationship wrong.

AI is not nuclear weapons, and the analogy breaks if pushed too far. A nuclear weapon does not reason with its operator, discover new science or help design the next nuclear weapon. But the nuclear age demonstrated something important. Greater capability can force human beings to invent forms of coordination that were previously unnecessary.

Then there is the money.

The companies building frontier artificial intelligence have become some of the most valuable enterprises ever created, or are moving rapidly in that direction. OpenAI reached a $500 billion valuation after current and former employees sold roughly $6.6 billion in shares. Anthropic is now discussing what could become the largest IPO in history, potentially valuing the company around $2 trillion.

That means some of the people closest to these companies may eventually become extraordinarily wealthy whether they remain inside them or leave.

That fact does not discredit their warnings. But it does complicate the story.

A researcher can sincerely fear the technology and still financially benefit from its extraordinary rise. An executive can genuinely believe development should slow while presiding over a company whose valuation depends partly on the belief that what it is building will transform civilization. Markets can simultaneously price in enormous future capability and remain uncertain about whether the underlying economics will ever justify the capital being poured into them.

Perhaps AI is exactly as transformative as its builders believe. Perhaps it is even more transformative. But human beings have been here before in another form. New technologies arrive carrying real breakthroughs, enormous possibility and narratives that begin outrunning what anyone can confidently know. Capital follows the story. Valuations climb. Talent moves toward the money. Skeptics are told they do not understand the future. Believers begin interpreting every new development as further evidence that the transformation is inevitable.

Sometimes they are right.

Sometimes the technology changes the world and the market still forms a bubble around it.

The internet was real. The dot-com bubble was real too.

Those statements do not contradict one another.

Artificial intelligence may be producing extraordinary technological progress and extraordinary financial reflexivity at the same time. Rising valuations attract more capital. More capital funds larger models, more data centers and higher compensation. Those investments create more visible capability, which strengthens the story that still greater capability is imminent. That story attracts more capital.

The market can begin participating in the reality it is predicting.

None of that tells us the boom will collapse. Today's frontier companies are generating real revenue, selling useful products and producing capabilities that did not exist several years ago. There are serious reasons to believe this technological shift is fundamentally different from many speculative manias. There are also extraordinary amounts of money moving through an ecosystem whose final economics remain unresolved. AI-related spending is expected to approach $800 billion in 2026 and more than $1 trillion in 2027, enough that even modest changes in expectations could move markets far beyond the AI companies themselves.

So perhaps the more honest position is not doom, optimism or dismissal. Perhaps it is uncertainty.

What if we are trying to understand artificial intelligence from inside one of the largest financial, technological and psychological events of our lifetime?

That makes discernment harder. It also makes discernment more important.

Beyond Survival

Whether artificial intelligence becomes the largest technological transformation in human history, the largest speculative financial narrative of our lifetime, or some complicated mixture of both, one thing already seems difficult to argue with.

A great deal of change is coming.

Work will change. Education will change. Science, business, art, medicine, warfare, communication and the distribution of knowledge will change. Some of those changes will arrive through artificial intelligence itself. Others will come from what human beings do in response to it.

Novelty is increasing.

That matters because rapid novelty and change place their own demands on cognitive flexibility. Perhaps that is where the conversation needs to become larger than both the dreams and the nightmares we are projecting onto the technology. The doom story asks how we prevent AI from destroying us. The boom story asks how much value AI can create. The bubble story asks whether we are collectively overestimating what it can become.

All three are legitimate questions.

None of them is large enough.

There is another question underneath them:

What kind of world are we trying to create with this increase in capability?

Protection matters. Safety matters. Alignment matters. Regulation matters. A technology capable of affecting billions of people should not be developed carelessly. But protecting humanity is not the same thing as supporting humanity.

Survival is the oldest directive in the nervous system. Detect the threat. Secure the resources. Defend the territory. Preserve the organism. Those capacities brought us here. They are also embedded in much of the response to artificial intelligence now. We worry that another company will get there first. Another country will get there first. The machine will acquire resources. The machine will preserve itself. Jobs will disappear. Power will consolidate. We need safeguards. We need dominance. We need control.

Some of those concerns are justified.

But if the developmental challenge of this moment is partly that our capability has outgrown the structures through which we have historically exercised it, then organizing the future entirely around survival may reproduce the very limitation we are trying to overcome.

The question cannot only be how we keep AI from harming humanity. It also has to become how this capability might support human development. Could it deepen access to education rather than merely make information cheaper? Could it expand medical care rather than simply make healthcare companies more efficient? Could it return time to human beings instead of simply increasing the amount of productivity expected from them? Could new intelligence strengthen human creativity rather than convince people their creativity is obsolete? Could it help us understand one another better rather than simply become more effective at persuading, targeting and manipulating one another? Could the extraordinary abundance potentially created by intelligent machines alter our relationship with scarcity rather than concentrate abundance among those who already possess the greatest leverage?

Those are not technological questions alone.
They are questions of values, meaning and development.

The machine cannot decide what a flourishing human life should be for us without us first surrendering something extraordinary: our responsibility to determine what we are trying to become.

Artificial intelligence does not only confront humanity with the possibility of something going terribly wrong. It confronts us with possibility itself.

Danger asks us to survive. Possibility asks us to choose.

The Threshold

This is where I move from what I can observe to how I have come to understand it.

I have come to see this pressure as evolutionary. I do not mean that I can prove the universe has a predetermined plan for human consciousness. I cannot. Evolution does not owe us enlightenment, and history gives us no guarantee that greater danger produces greater wisdom.

I mean something simpler.

The math increasingly has to add up.

As capability rises, the cost of operating from developmental structures unable to hold that capability rises with it. An identity organized entirely around me has limitations. An identity capable of including a family can coordinate realities the isolated individual cannot. Tribes organize forms of complexity an individual cannot. Nations coordinate at scales tribes cannot. International systems become necessary when problems cross borders.

Each wider order does not erase the one beneath it. I do not stop being an individual when I become part of a family. A nation does not cease to exist because humanity shares an atmosphere.

Development expands what we are capable of holding at once.

That capacity may no longer be optional.

We are building technologies whose consequences operate globally while responding through institutions still heavily organized around competition between parts. We are economically, biologically, technologically, and ecologically entangled in ways that make complete isolation impossible, yet fear can return us almost instantly to ancient distinctions between them and us.

Technology can also develop much faster than the human beings and institutions responsible for it. A model can be upgraded in months. Political systems may require decades to change. Culture moves unevenly. Human nervous systems carry responses forged under conditions radically different from the environment our creativity has now produced.

Human capability is climbing the ladder faster than human consciousness. Eventually something has to give.

There is a danger in talking at this scale. We start speaking about humanity, civilization, nations, superintelligence and evolution until the problem becomes so enormous that nobody can find themselves inside it.

But there is no abstract place called humanity where consciousness develops.

It happens here.

A nation cannot investigate its fear. A corporation cannot recognize its projection. Humanity cannot turn within.

A person can.

Someone decides whether to release a model. Someone writes the regulation. Someone chooses whether to treat another nation as an adversary. Someone decides where the capital goes and whether a particular risk is worth taking. Someone interprets the threat. Someone feels the contraction in their own body when the future suddenly looks dangerous.

The species-level problem becomes intimate whether we want it to or not.

That is why I am not interested in using shadow work to explain away catastrophic risk. I am interested in what becomes possible when we can face catastrophic risk without becoming completely organized by it.

Can I distinguish what I know from what I fear? Can I recognize the story my fear is building around the facts? Can I remain capable of curiosity while taking danger seriously? Can I see an adversary clearly without requiring them to carry everything I refuse to recognize in myself? Can I tolerate enough uncertainty to encounter something genuinely new? Can I participate in a larger identity without pretending my smaller identities no longer matter?

These sound like personal questions because they are.

They are also the raw material from which boards, laboratories, governments and civilizations make decisions.

For generations, consciousness development could be treated as a philosophical interest, spiritual pursuit or personal choice. Something we explored because life hurt, because a relationship failed, because an old pattern stopped working or because some deeper part of us wanted to discover what else was possible.

Increasingly, I wonder whether that framing is becoming obsolete.

Perhaps human development is becoming infrastructure.

Not a luxury we add after solving the serious problems, but part of what determines whether we can solve them at all.

We will need engineers capable of building safer systems. We will need independent evaluation, thoughtful regulation and institutions capable of responding at technological speed. We will need governments willing to protect legitimate national interests while recognizing shared ones. We will need agreements between countries that do not trust one another. We will need scientists, philosophers, business leaders, citizens and disciplines that may not yet exist.

None of that is replaced by turning within.

But none of it escapes the human being doing it either.

The people building the safeguards, writing the regulations, allocating the capital, commanding militaries and negotiating with competitors still arrive carrying nervous systems, identities, histories, ambitions, wounds, fears, genius and shadow. As our power increases, what comes through us carries greater consequence.

Our fear, greed and tribalism matter more.

So do our imagination, creativity, compassion and capacity for relationship.

Our ability to recognize the difference between a danger and everything we have attached to that danger matters more because we now possess more ways to act upon the distinction.

That is the part of the AI problem no technical system can solve for us.

AI may eventually become conscious. It may not. Superintelligence may arrive rapidly, slowly or in some form none of today's forecasts accurately imagine. The catastrophic scenarios may prove prescient. They may prove wildly exaggerated. Today's valuations may eventually look conservative, or we may look back on this period and recognize the unmistakable psychology of a bubble wrapped around a genuinely transformative technology.

I don't know.

That uncertainty is not a weakness in the argument. It may be the most important fact we have.

Something has already changed.

Humanity is becoming powerful enough that our developmental limitations are no longer confined to the private consequences of an individual life. They can travel through technologies and institutions capable of touching millions and eventually billions of people.

The distance between our capability and our capacity to hold it is becoming consequential.

I have come to believe that distance is the real threshold.

Perhaps evolution does not promise that we cross it. Perhaps it simply keeps presenting us with conditions that make the limitations of our current way of being harder to ignore.

Artificial intelligence may just be one of those conditions.

We should build safeguards. We should aggressively test these systems. We should regulate where regulation is warranted, strengthen institutions and take seriously the warnings coming from people closest to the technology. We should also remain skeptical enough to examine financial incentives, geopolitical narratives, technological hype and our own projections onto what we think AI will become. None of that is replaced by inner work. 

But eventually every system, regulation, model, market and institution passes through human hands. If we organize this moment only around survival, we may succeed at protecting humanity without ever asking what humanity is being protected for. Our capability has changed. The question now is not only whether our capacity to hold it can change with it. It is can we become conscious enough to decide what we want all of this capability to serve.