An artificial intelligence researcher at Anthropic has resigned after raising serious concerns about the speed at which powerful AI systems are being developed and the risks they could create.
Jacob Coxon, a 27-year-old researcher at Anthropic, said he was leaving the company because he no longer wanted to take part in what he sees as a dangerous race to build increasingly powerful artificial intelligence.
Coxon warned that AI companies are moving toward systems that could become capable of improving themselves with less human involvement. According to him, this could make future AI much harder to understand, control or stop if something goes wrong.
He also raised concerns about competition between major AI companies. When several companies are trying to develop the most advanced AI first, each business may feel pressure to move faster to avoid falling behind. Coxon believes this competition could make it difficult for individual companies to slow development even when researchers see serious safety risks.
His resignation is notable because Anthropic has built much of its identity around developing AI responsibly and researching ways to keep advanced systems aligned with human goals.
The concerns are not limited to Coxon. Evan Hubinger, who leads alignment research at Anthropic, has also publicly discussed the possible dangers of extremely advanced AI. He said he believes there is a greater than 10% chance that advanced AI could cause catastrophic harm to humanity within the next decade.
These estimates remain predictions rather than confirmed outcomes. Today’s AI systems do not have the abilities described in many of these worst-case scenarios. However, researchers are increasingly focused on what could happen if future models become more autonomous, learn to improve their own abilities or operate with less human supervision.
One major challenge is known as AI alignment. It focuses on making sure an AI system continues to follow human goals and safety rules even as its abilities become more powerful. Researchers are still working on reliable ways to guarantee that highly advanced AI systems will remain under human control.
Coxon’s departure adds to a wider debate inside the AI industry about whether development is moving faster than safety research. Some experts are calling for stronger testing, clearer rules, greater transparency and cooperation between governments and AI companies.
For companies such as Anthropic, OpenAI and other developers of advanced AI models, the challenge is becoming increasingly important. They want to continue improving AI because of its potential benefits in science, software, healthcare, business and many other areas, while also making sure increasingly capable systems do not create risks that society is unable to manage.
Coxon’s warning highlights the difficult balance facing the AI industry: continuing rapid innovation while making sure safety protections can keep up with the technology.
