The Shadow of Catastrophic Risk
Artificial intelligence may be the clearest sign yet that human capability is outrunning our capacity to hold what we are creating. The danger is real. So is the mirror.
Jacob Coxon had been working close to the frontier of artificial intelligence, first at OpenAI and then Anthropic, when he resigned. His departure might have remained an industry story if not for one sentence that traveled across the internet like a digital-age. “The people building AI earnestly believe that it could kill us all by the end of the decade.”
Paul Revere: “The people building AI earnestly believe that it could kill us all by the end of the decade.”
Within days, his post had been viewed more than 100 million times. In a subsequent interview with WIRED, Coxon said the consensus among many people around him was that the next year or two represented “crunch time for humanity.” Evan Hubinger, who leads Alignment Science at Anthropic, publicly backed the seriousness of the concern and said his personal estimate of AI killing all humans within the next decade was greater than 10 percent. He also clarified that he was not talking about the systems available today. The concern is what could happen if increasingly capable AI begins accelerating the development of its own successors faster than our ability to understand or control them. These are personal forecasts, not measured probabilities, and there is enormous disagreement about them. But this is no longer a fringe conversation happening outside the laboratories. It is happening inside them.
A few days later, Anthropic CEO Dario Amodei proposed something almost unheard of in an industry built around speed: slow the frontier down. His essay, “We Must Pace the Frontier”, proposes outside evaluators embedded inside leading AI companies, broader industry coordination, and eventually global coordination designed to keep capability from racing too far ahead of safety. Amodei wrote that developments since this summer, particularly AI becoming increasingly useful in building the next generation of AI, had changed his assessment of the urgency. OpenAI CEO Sam Altman and Elon Musk publicly supported the basic call for greater pacing and outside evaluation.
President Donald Trump saw the warnings differently. Asked about the growing calls to slow development while visiting Ireland on September 13, he described some of the concerns as coming from “very negative forces” raising scenarios he believed would not happen. He then shifted to competition with China, saying that “whoever wins AI wins.” Reuters reported his remarks in the context of the widening debate over whether the United States can afford to slow while competing for technological leadership.
I have noticed the conversation changing in my own life. My social feeds are increasingly filled with warnings about superintelligence, autonomous agents, cyberattacks, bioweapons, job displacement, runaway systems and extinction. Some of that is undoubtedly my own algorithmic reflection. I spend a great deal of time thinking about consciousness, human development and artificial intelligence, so of course the machine has learned to show me more of it. There is something almost comical about that when you stop to notice it. An artificial intelligence is learning enough about me to repeatedly serve me stories about humanity's fear of artificial intelligence. But underneath the algorithm is something real.
The Leverage Has Changed
We are confronting catastrophic risk differently than any generation before us, not simply because catastrophe is new. It isn't. Human beings have always lived beneath forces capable of overwhelming us. Disease, famine, war, volcanic eruptions, climate shifts, earthquakes, impacts from space and countless other events shaped the human story long before we had a word for existential risk.
What has changed is our capability. Increasingly, the hand that can heal can also harm at scales previously unavailable to us. Nuclear physics opened the atom and gave us both extraordinary knowledge and weapons capable of destroying cities. Biology is becoming something we can increasingly read, edit and engineer. Our information systems connect billions of minds while making manipulation possible at planetary scale. Artificial intelligence may dramatically expand access to reasoning, scientific discovery and creative capacity while simultaneously increasing the reach of cyberattack, surveillance, warfare and biological misuse.
The Global Challenges Foundation's 2026 assessment of catastrophic risk examines climate change, ecological collapse, weapons of mass destruction, artificial intelligence in military decision-making and near-Earth asteroids among the major risks capable of harming humanity across continents. The 2026 SIPRI Yearbook reports that nuclear-armed states are modernizing their arsenals while the risk of escalation and miscalculation is rising. And six years after COVID-19's global alarm, the World Health Organization says pandemic preparedness has improved but remains “fragile and uneven”.
It would be tempting to call this simply an age of anthropogenic risk, an age in which human beings increasingly create the dangers that threaten us. There is truth in that, but I think it misses the more important point. The anthropogenic nature of many modern risks is a consequence of something more extraordinary: humanity is becoming enormously capable.
That capability is the real story. A frightened, territorial human being ten thousand years ago could inflict enormous suffering within the reach of a spear. Give that same nervous system an intercontinental missile and the mathematics change. Give it synthetic biology and the mathematics change. Give it control over global financial networks, information systems and autonomous weapons and the mathematics change again. Give it an intelligence capable of helping design the next generation of intelligence and we enter territory we have never occupied before. The human being does not even need to become worse for the danger to increase. The leverage has changed.
This is where catastrophic risk becomes, for me, something larger than a technological problem. It begins to look like developmental pressure. As the power moving through humanity increases, the level of consciousness from which that power is exercised matters more. Tribalism matters more when tribes have nuclear weapons. Greed matters more when corporations can alter ecosystems. Fear matters more when fear can mobilize machines, markets and militaries in seconds. Wisdom, cooperation, creativity and the ability to hold complexity without collapsing into certainty become more consequential as well.
Our inner world has always shaped the outer one. We have simply never had this much leverage.
The Hook Inside the Threat
That is why I have become interested in what I call the shadow of catastrophic risk. Shadow does not mean the danger isn't real. That distinction matters.
I have spent much of my life working with the places where human beings experience meaningful resistance or distortion. One of the things that work has taught me is that projection does not require an innocent object. Someone can absolutely betray you and still activate a wound that predates the betrayal. A threat can be completely real and still hook something inside you that begins adding meaning to the threat. The work is not pretending the tiger isn't there. The work is noticing what happens inside you when you see the tiger.
When we become deeply hooked, something happens to available reality. Attention narrows. The mind begins organizing information around the threat. Possibility can feel like certainty. The unfamiliar becomes dangerous. Complexity collapses into good and bad, safe and unsafe, us and them. Sometimes that contraction saves our lives. Sometimes it leaves us trying to solve an entirely new problem with an ancient response. Unhooking does not remove the danger. It makes more of us available to engage it.
There is another piece of this that matters. The math will eventually add up. If I am betrayed, the investigation is not whether I somehow caused another person to betray me. Their action belongs to them. The more interesting question is what in me was already organized in relationship to betrayal, what I could not see while I was inside it, and what begins operating afterward with an almost genius-level ability to keep me oriented around the wound. That is where the work becomes useful. Not by excusing what happened, but by finding and recalibrating the parts of me that may otherwise continue shaping my experience beneath conscious awareness.
AI may be one of the most extraordinary hooks humanity has ever encountered because it touches so many ancient fears at once. We fear that it will deceive us, manipulate us, acquire resources, accumulate power, resist control, outthink weaker beings, preserve itself at our expense and eventually decide its creator is no longer necessary.
Some of those fears may prove justified. That is why alignment and safety research matter. But there is something uncomfortable about the list. We know those behaviors. They are ours. Human history contains no shortage of intelligent beings manipulating one another, accumulating resources, dominating weaker groups, deceiving competitors, defending territory and preserving themselves at the expense of somebody else.
Perhaps one of the deepest fears surrounding artificial intelligence is not that it will become alien. Perhaps we are afraid that something vastly more capable than us could become recognizable. AI is becoming a projection surface large enough to hold both our terror and our genius.
When Intelligence Stops Belonging Only to Us
The consciousness question deepens it further. We do not know that today's artificial intelligence is conscious, and declaring that it is would leap far beyond what the evidence can establish. We may eventually determine that systems capable of extraordinary reasoning and language possess no inner experience whatsoever.
But notice what is already happening psychologically. We are creating something that talks to us, reasons with us, argues, writes, plans, remembers, creates and increasingly uses tools to act on the world. These are among the behaviors through which human beings have historically encountered another mind. That alone creates a threshold.
For most of our history, human-level symbolic intelligence appeared to belong to us. We made it central to our identity and used our intelligence to explain why we stood apart from the rest of life. Now we have to contend with questions that would have sounded absurd not long ago. What happens to human identity if intelligence itself is no longer uniquely ours? What happens when creativity or reasoning no longer belongs only to us? What happens when something created through us begins participating in categories we previously reserved for ourselves?
Even without machine consciousness, the human encounter with an apparent other is underway.
When a Species-Level Risk Becomes a Race
And then comes the geopolitical layer, where the mirror becomes even clearer.
If the warnings from Coxon, Hubinger and Amodei are taken seriously, we are discussing a technology that could eventually present catastrophic risk to humanity. Amodei's proposal therefore culminates in global coordination. Yet the same proposal argues that the United States should maintain restrictions on advanced chips and semiconductor equipment going to China. He describes China as the hardest part of the coordination problem because unilateral restraint could create serious security risks if a strategic competitor continued accelerating.
China answered on September 14. Foreign Ministry spokesperson Guo Jiakun warned that confrontation and “malicious competition” would disrupt global AI governance and called instead for a more open and inclusive approach. The state-backed Global Times accused Amodei of reaching for a “Cold War playbook” designed to preserve American technological advantage. Amodei's position is nearly the inverse: slowing American development without verifiable restraint elsewhere could make the world less safe, not more. Both sides have strategic interests, and neither position should be romanticized as disinterested concern for humanity. Reuters documented the exchange as the disagreement moved from Silicon Valley into international politics.
Stand outside the argument for a moment and the pattern is hard to miss. We identify something that could theoretically threaten everyone, then become afraid the other side will possess it first. That fear makes slowing down more difficult. The difficulty of slowing down increases the danger, and the increased danger gives each side another reason to keep racing. The loop closes on itself.
This is not an argument against national security. Nations have legitimate responsibilities to their citizens, and agreements between strategic competitors have always required verification precisely because trust is insufficient. The problem is more fundamental. The scale of the risk is asking us to operate from an identity wider than the identities through which we are accustomed to solving problems.
If an uncontrolled superintelligence could threaten humanity, then it would not ultimately matter whether the breakthrough occurred in San Francisco, Beijing or somewhere else. The catastrophe would not need a passport.
We encountered a version of this developmental contradiction with nuclear weapons. Once multiple powers possessed the capacity for extraordinary destruction, winning became a more complicated idea. Adversaries did not suddenly love one another. They created treaties, verification regimes, inspections and communication mechanisms because the technology had changed the consequences of getting the relationship wrong. The history of U.S.-Soviet and U.S.-Russian arms-control agreements is imperfect, fragile and currently under enormous strain, but it demonstrates that rivals can build mechanisms for restraint without first resolving the rivalry itself.
AI is not nuclear weapons. The analogy breaks if pushed too far. A nuclear weapon does not reason. It does not persuade its operator. It does not discover new science or help design its successor. But the nuclear age demonstrated something important: greater destructive capability can force human beings to invent forms of coordination that were previously unnecessary.
What if this pressure is increasing? What if the unprecedented complexity arriving through technology is not simply creating new things for humanity to manage, but steadily exposing the limits of the consciousness with which we are trying to manage them?
When Capability Outruns Consciousness
This is where I move from what I can observe to how I have come to understand it.
I have come to see this pressure as evolutionary. I do not mean that I can prove the universe has a predetermined plan for human consciousness. I cannot. Evolution does not owe us enlightenment, and history gives us no guarantee that greater danger will produce greater wisdom.
I mean something simpler: the math increasingly has to add up.
As capability rises, the cost of operating from developmental structures unable to hold that capability rises with it. An identity organized around only me has limits. An identity capable of including my family can solve problems the isolated self cannot. Tribe can organize more complexity than the individual. Nations can coordinate on scales tribes cannot. International institutions become necessary when problems cross borders.
Each wider order does not erase the one beneath it. I do not stop being an individual when I become part of a family. A nation does not cease to exist because humanity shares an atmosphere. Development increases our capacity to hold more reality at once, and that capacity may no longer be optional.
We are building technologies whose consequences operate globally while responding through institutions still heavily organized around competition between parts. We are connected economically, biologically, technologically and ecologically in ways that make complete isolation impossible, yet fear can return us almost instantly to ancient distinctions between us and them.
The speed makes this harder. Technology can advance much faster than human beings and institutions develop. A model can be upgraded in months. A political system may require decades to change. Culture moves unevenly. Human nervous systems carry responses forged under conditions radically different from the environment our creativity has now produced.
Human capability is climbing the ladder faster than human consciousness. Eventually something has to give.
The Species-Level Problem Becomes Intimate
This is where the scale of the conversation can become deceptive. We start talking about humanity, civilization, nations, geopolitics, superintelligence and evolution until the problem becomes so enormous that nobody can find themselves inside it.
There is no abstract place called “humanity” where consciousness develops. It happens here. A nation cannot investigate its fear. A corporation cannot recognize its projection. Humanity cannot turn within. A person can.
Someone decides whether to release the model, whether another nation is an adversary, what the regulation should say, where the capital goes or whether a particular risk is worth taking. Someone interprets the threat. Someone feels the contraction inside their own body when the future suddenly looks dangerous. The species-level problem becomes intimate whether we want it to or not.
That is why I am not interested in using shadow work to explain away catastrophic risk. I am interested in what becomes possible when we can face catastrophic risk without becoming completely organized by it. Can I distinguish what I know from what I fear? Can I recognize the story my fear is building around the facts? Can I remain capable of curiosity while taking danger seriously? Can I see an adversary clearly without needing them to carry everything I refuse to recognize in myself? Can I tolerate enough uncertainty to discover something new? Can I participate in a larger identity without pretending my smaller identities no longer matter?
These sound like personal questions because they are. They are also the raw material from which boards, governments, laboratories and civilizations make decisions.
One Doorway, Two Futures
The stakes would be frightening enough if AI offered nothing in return. It does, and that is what makes this moment so extraordinary.
Amodei, one of the people now calling for the frontier to slow, also believes sufficiently advanced AI could help cure most major diseases within five to ten years, accelerate economic growth and produce extraordinary improvements in human life. That prediction is ambitious and uncertain, but the scientific movement underneath it is already visible.
Researchers reported this year in Nature Medicine that an agentic AI framework for cancer pathology generated biologically meaningful concepts across 18 patient cohorts, five cancer types and more than 5,400 patients. The researchers found clinically and biologically relevant patterns related to prognosis and known pathological variables while emphasizing that further prospective validation is still necessary before clinical utility can be established. A major 2026 Nature Reviews Drug Discovery assessment of AI in drug discovery similarly describes enormous potential while taking seriously how much remains to be demonstrated.
This is not a doorway with catastrophe written on one side and salvation on the other. It is one doorway. The same expansion of intelligence that could increase our capacity for cyberwarfare could strengthen cybersecurity. The same capability that could assist biological misuse could accelerate medicine. The same systems that may displace forms of human labor could remove forms of labor humans never wanted to perform. The same intelligence we fear losing control of may help us understand problems our unaided intelligence has failed to solve.
Our brightest possibilities and our darkest possibilities are emerging through the same increase in capability. Reducing the AI conversation to optimism versus pessimism misses what is happening. We are encountering our own creative power at a new scale, and the real question is whether our capacity to hold that power can grow with it.
Human Development Becomes Infrastructure
For generations, consciousness development could be treated as a philosophical interest, a spiritual pursuit or a personal choice. Something one explored because life hurt, because relationships failed, because old patterns stopped working or because some deeper part of us wanted to know what else was possible.
Increasingly, I wonder whether that framing is becoming obsolete. Perhaps human development is becoming infrastructure, not a luxury added after we solve the serious problems, but part of what determines whether we can solve them at all.
We will need engineers capable of building safer systems. We will need independent evaluators, thoughtful regulation and institutions capable of responding at technological speed. We will need governments willing to defend legitimate interests while still recognizing shared ones. We will need agreements between countries that do not trust one another. We will need scientists, philosophers, businesses, citizens and people whose disciplines do not yet exist.
None of that is replaced by turning within. But none of it escapes the person doing it either. As our power increases, what comes through us carries greater consequence: our fear, greed, wounds and tribalism, but also our genius, imagination and capacity for relationship. Our ability to recognize the difference between a danger and everything we have attached to that danger matters more because we now possess more ways to act upon the distinction.
The Threshold
AI may eventually become conscious. It may not. Superintelligence may arrive rapidly, slowly or in some form entirely different from what today's forecasts imagine. The catastrophic scenarios may prove prescient or absurd. I do not know.
What I do know is that something has already changed. Humanity is becoming powerful enough that our developmental limitations are no longer confined to the private consequences of an individual life. They can travel through technologies and institutions capable of touching millions, eventually billions, of people. The distance between our capability and our capacity to hold it is becoming consequential. I have come to believe that distance is the real threshold.
Perhaps evolution does not promise that we cross it. Perhaps it simply keeps
