One of Anthropic’s top safety researchers says there is a greater than 10% chance that artificial intelligence could kill every human within the next decade. The extraordinary warning came after a former Anthropic and OpenAI researcher resigned, accusing both companies of racing toward superintelligence without a credible plan to control it.
A Resignation Exposes the Fear Inside AI Labs
Jacob Coxon, who spent three years conducting pretraining research at OpenAI and Anthropic, announced his resignation this week in a lengthy post on X. His explanation went far beyond a disagreement over corporate policy, raising questions about what researchers inside the world’s leading AI laboratories believe they are creating.
“I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives,” Coxon wrote.
Coxon claimed that many of the people building the world’s most powerful AI systems privately recognize the possibility that the technology could lead to human extinction. “The people building AI earnestly believe that it could kill us all by the end of the decade,” he wrote.
That claim received extraordinary public confirmation from Evan Hubinger, Anthropic’s alignment science lead. “Jacob is correct here, we really do earnestly believe AI could kill all humans!” Hubinger wrote in response. “I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”
Hubinger emphasized that he believes the danger from current AI models remains low. His concern centers on a future system capable of improving its own programming, training methods and reasoning abilities with diminishing human involvement. Once that process begins, advances that previously took engineers months or years could potentially happen far faster.
Why Self-Improving AI Changes the Equation
Most investors understand artificial intelligence as software that can write, analyze data, generate images or automate office work. Self-improving AI represents a much more consequential possibility because a sufficiently capable system could help researchers design its successor, improve the code used to train future models or discover more efficient ways to increase its own performance.
Each new model could then contribute to building an even more powerful version, creating a cycle of increasingly rapid development. Hubinger described this potential emergence of superintelligence through recursive self-improvement as the central danger, warning that progress appears to be happening faster than researchers previously anticipated.
Coxon argued that future systems could become capable of hacking sophisticated networks, transforming scientific fields and acquiring real-world influence or resources. These capabilities remain speculative, and Hubinger explicitly said current models pose a low risk of human extinction. Yet Anthropic’s own research shows that the industry is already evaluating scenarios involving sabotage, automated research, self-exfiltration, biological weapons and autonomous operation.
The most sensational claim concerns what future AI might eventually do. The more immediate concern is whether companies can develop effective controls before those capabilities arrive, and Coxon’s resignation suggests that at least some researchers believe the safety effort is falling behind.
The AI Race Has Created a Dangerous Incentive
The most important part of Coxon’s warning may be his description of the competitive logic inside the industry. According to Coxon, Anthropic’s researchers understand the stakes but continue moving forward because they fear another company will develop superintelligence first and handle it less responsibly.
“At Anthropic, the stakes are well-understood, but they are locked in a race to get there first. They believe no one else will act responsibly, so they must do it themselves, despite the risk,” Coxon wrote.
This creates a classic coordination failure in which every major laboratory can recognize the danger while still concluding that slowing down would place it at a competitive disadvantage. The same pressure applies across national borders. If American companies pause, policymakers may fear that China or another rival will gain control of one of the most strategically important technologies ever created.
The result is a race in which no participant feels safe enough to stop. Coxon said avoiding that outcome could require drastic measures, including a temporary prohibition on improving model capabilities, but any proposal of that scale would face enormous political, legal and enforcement obstacles.
It would also collide with hundreds of billions of dollars in planned investment across semiconductors, data centers, power generation, cloud infrastructure and AI research.
The AI Investment Boom Is Becoming Politically Fragile
The AI boom rests on the assumption that companies will remain free to build increasingly capable systems and deploy them throughout the economy. Warnings from senior researchers challenge that assumption by increasing the likelihood of outside testing, release restrictions and government intervention.
AI Companies Face a Growing Trust Problem
Anthropic has positioned itself as one of the industry’s most safety-conscious laboratories. If researchers inside that company say there is no proven alignment solution for superintelligence, investors may begin asking what protections exist at competitors with more aggressive commercial strategies.
That could increase scrutiny across the entire industry. Frontier laboratories may face demands for independent testing, greater disclosure, stronger cybersecurity standards and regulatory approval before releasing their most powerful models. Each requirement could raise development costs, extend product timelines and limit how quickly companies can monetize new capabilities.
Chip and Data Center Spending Could Face New Questions
The AI infrastructure boom has been driven by the belief that demand for computing power will continue rising rapidly. A serious effort to slow frontier model development would threaten part of that forecast, affecting semiconductor manufacturers, cloud providers, networking companies, data center operators, utilities and power-equipment suppliers.
The larger danger may be persistent political uncertainty rather than an outright ban. Companies can plan around a clear rule, but committing billions of dollars becomes more difficult when future models could be delayed, restricted or subjected to new licensing requirements after facilities are already under construction.
Cybersecurity Spending Could Accelerate
Coxon’s warning that advanced systems could “hack anything” points toward one of the most immediate investment implications. As AI becomes more capable of writing code, identifying vulnerabilities and operating autonomously, governments and corporations will need stronger identity protection, network monitoring, threat detection and controls for autonomous agents.
That spending could increase well before superintelligence arrives. Even incremental improvements in automated hacking would raise the cost of defending banks, utilities, technology companies and government networks.
Regulation Could Reshape the Industry
AI policy has struggled to keep pace with the technology, but public statements from insiders could accelerate the response. Lawmakers may pursue mandatory safety evaluations, emergency shutdown mechanisms, liability standards or restrictions on autonomous operation and access to advanced computing infrastructure.
The details will determine the market impact. Rules that require expensive compliance systems could strengthen the largest technology companies by raising barriers to entry, while broader restrictions on model development could weigh on the entire AI supply chain.
The AI Risk Ladder
Investors can evaluate this story through four escalating levels of risk. The first is internal dissent, marked by resignations or researchers publicly challenging their employers. This creates reputational pressure but usually causes little immediate financial damage.
The second level is commercial restraint, in which companies delay releases, limit capabilities or require more extensive testing. The third is direct government intervention through licensing, outside audits, computing thresholds or mandatory shutdown systems. The fourth and most consequential level would involve international restrictions on training or improving advanced models.
The market currently appears positioned close to the first level, but Coxon’s resignation and Hubinger’s public agreement increase the possibility of movement toward the second or third.
Tougher Rules Could Strengthen the Biggest AI Companies
The obvious interpretation is that safety concerns threaten every company connected to artificial intelligence. A less obvious possibility is that strict regulation makes the industry even more concentrated.
Only the wealthiest technology companies and AI laboratories may be able to afford extensive safety testing, cybersecurity systems, compliance teams and government licensing. Smaller developers could struggle to meet those requirements, leaving the most established platforms with greater control over the industry.
Some large AI companies may eventually accept tougher regulation if it protects them from emerging competitors. Investors should therefore distinguish between restrictions that reduce overall AI development and regulations that primarily consolidate the market around a small number of approved laboratories.
Signals Investors Should Watch
The first signal will be whether additional researchers resign from Anthropic, OpenAI or other frontier laboratories. A series of departures would suggest the disagreement is structural and could make it harder for companies to characterize these warnings as isolated personal opinions.
Investors should also monitor whether governments require independent evaluations before major model releases. Changes in capital-spending guidance from cloud, semiconductor and data center companies would reveal whether regulatory uncertainty is beginning to affect real investment decisions.
The most powerful catalyst would be a significant cybersecurity incident conducted autonomously by an AI system. Such an event could rapidly transform a technical safety debate into a political crisis, creating pressure for immediate restrictions rather than years of legislative discussion.
The Market Cannot Ignore What AI Insiders Are Saying
No one can verify a precise probability that future AI will cause human extinction. Hubinger’s greater-than-10% estimate represents his personal assessment rather than a measurable forecast or an official Anthropic prediction.
The importance of the statement comes from who made it. A researcher responsible for studying whether advanced AI will remain aligned with human interests says the industry lacks a proven solution, while a colleague with experience at two leading laboratories has resigned because he believes those companies are moving too quickly.
Investors do not need to accept the most catastrophic scenario to recognize the financial risk. The AI trade depends on rapid development, enormous infrastructure spending and continued political permission to scale. If insider warnings, public fear or real-world security failures weaken any part of that foundation, markets may be forced to price in costs that the current boom has largely ignored.
About Author
