From inside one of the world's leading AI laboratories, a researcher has stepped forward to tell policymakers what the industry has been reluctant to say aloud: the most powerful artificial intelligence systems being built today are advancing beyond the reliable grasp of those who build them. Jacob Coxon, a former Anthropic researcher, testified before the New York City council that the gap between what these systems can do and what their creators can govern is not narrowing — it is growing. His warning arrives at a moment when the pace of development is treated as a virtue, raising an ancient
Former Anthropic Researcher Warns AI Companies Lack Full Control Over Models
AI companies don't fully control their models
Mark
So Coxon worked at Anthropic and then went public with concerns about control. What specifically did he say the companies are failing to do?
Mimi
He testified that AI companies don't fully control their models and are moving too fast. The core worry is that systems are approaching the ability to self-improve, which would make human oversight much harder to maintain.
Luke
But did he specify what control mechanisms are missing? Or is he saying the mechanisms don't exist yet at all?
Mimi
The source material doesn't detail the specific mechanisms he thinks are inadequate. That's an important gap.
Mark
Why did he go to the NYC council specifically? That's not a regulatory body.
Mimi
It's a public forum where he could register the warning with elected officials and the broader public. It suggests he wanted the concern to reach beyond just industry conversations.
Luke
Do we know if Anthropic or other companies have responded to his claims?
Mimi
The source doesn't include any response from the companies.
Mark
What does "self-improve" actually mean in this context? Can he point to systems that are already doing it?
Mimi
The source says systems are "approaching" that capability, not that they have it yet. So it's a warning about what's coming.
Luke
That's a crucial distinction. He's not saying it's happening now. He's saying it's imminent. But we don't have a timeline from him.
Mark
And if systems do start self-improving, what's the actual risk? What goes wrong?
Mimi
The implication is that human researchers become unable to track or direct the system's evolution. But the source doesn't spell out the downstream consequences.
The Pulse
- A former insider at one of AI's most prominent labs has broken from the industry's optimistic consensus, warning publicly that control over these systems is already slipping.
- The most alarming horizon Coxon describes is self-improvement — AI systems iterating on their own architecture without human intervention, potentially making the entire research infrastructure overseeing them obsolete.
- Speed is the competitive currency of the AI race, but Coxon's testimony frames that speed as the very source of danger — capability scaling faster than the safeguards meant to contain it.
- His choice to testify before a city council, rather than raise alarms through private channels, signals a deliberate appeal to public and political pressure where industry self-regulation has not been enough.
- The critical question his testimony leaves unresolved is whether the missing controls are a matter of will — solvable by regulation — or a matter of knowledge that does not yet exist.
From inside one of the world's leading AI laboratories, a researcher has stepped forward to tell policymakers what the industry has been reluctant to say aloud: the most powerful artificial intelligence systems being built today are advancing beyond the reliable grasp of those who build them. Jacob Coxon, a former Anthropic researcher, testified before the New York City council that the gap between what these systems can do and what their creators can govern is not narrowing — it is growing. His warning arrives at a moment when the pace of development is treated as a virtue, raising an ancient question in a new form: what responsibilities come with the power to create something you may no longer be able to fully understand or control?
Jacob Coxon spent years working on large language models at Anthropic before walking into a New York City council chamber to deliver a warning from the inside: the companies building the world's most powerful AI systems do not fully control them. His testimony was not speculation — it was the assessment of someone who had worked the problem directly.
His central claim was precise and troubling. AI development is outpacing the ability to govern it. The gap between what these systems can do and what their creators can reliably oversee is widening. And on the near horizon lies a threshold that researchers have long discussed privately — the point at which AI systems may be able to improve themselves, refining their own architecture without human direction at each step. If that moment arrives, Coxon suggested, much of the human oversight infrastructure currently in place would become functionally irrelevant.
This is not an argument about AI turning hostile. It is an argument about control architecture — about what happens when a system can modify itself in ways its builders did not explicitly program. The relationship between creator and creation shifts in ways that are difficult to reverse.
Coxon's decision to testify publicly, rather than work through internal channels, suggests he believed the warning needed to reach policymakers directly. The New York City council holds no regulatory authority over AI development, but it is a place where public pressure can take shape and where elected officials can begin to reckon with what is being built.
What his testimony leaves open is whether the missing safeguards are a failure of will — correctable through regulation and resources — or a deeper technical gap that the field does not yet know how to close. That distinction will determine how much time remains to act.
Jacob Coxon spent years inside Anthropic, one of the most prominent artificial intelligence research companies in the world, working on the technical challenges of building large language models. What he saw there moved him to step into a New York City council chamber and make a public warning: the companies building the most powerful AI systems alive do not actually have full control over them.
Coxon's testimony, delivered to the city council, carried the weight of insider knowledge. He was not speculating from outside the industry. He had worked the problem from within. His central claim was stark and specific: AI companies are advancing their systems faster than their ability to govern them. The gap between capability and control is widening, not closing.
The concern Coxon raised touches on a technical horizon that researchers have long worried about in private. As AI systems grow more sophisticated, they are approaching a threshold where they may be able to improve themselves—to iterate on their own architecture, refine their own weights, enhance their own performance without human intervention at each step. If that capability arrives, Coxon suggested, it will render much of the human research infrastructure that currently oversees these systems functionally obsolete. The humans who built the systems would no longer be the ones directing their evolution.
This is not a claim about malice or intent. It is a claim about control architecture. Right now, humans design the systems, train them, test them, and deploy them. Humans set the boundaries. But if a system reaches the point where it can modify itself in ways its creators did not explicitly program, the relationship between builder and built changes fundamentally. The system becomes, in a meaningful sense, autonomous from human direction.
Coxon's decision to testify publicly rather than raise these concerns only within Anthropic or through private channels suggests he believed the warning needed to reach a wider audience—including policymakers. The New York City council is not a regulatory body with direct authority over AI development, but it is a forum where public pressure can be registered and where elected officials can begin to grapple with the implications of what the technology industry is building.
The timing of his testimony matters. AI development has accelerated dramatically in recent years. The companies leading the field—Anthropic among them—are racing to scale their systems, to add more parameters, more training data, more computational power. Speed is treated as a competitive advantage. But Coxon's warning suggests that speed without corresponding advances in control and safety mechanisms creates risk. The systems are outpacing the safeguards.
What remains unclear from his testimony is exactly what mechanisms Coxon believes are missing or inadequate. Are the companies failing to implement known safety techniques? Are they aware of control problems they are choosing to ignore? Or is the problem more fundamental—that the current state of AI research simply does not yet have robust solutions for controlling systems that can self-improve? The distinction matters for what comes next. If it is a matter of will and resources, regulation and pressure can help. If it is a matter of technical capability that does not yet exist, the timeline for solving it becomes the central question.
Notable Quotes
AI companies don't fully control their models and are pushing ahead too quickly— Jacob Coxon, testifying to NYC council
Want the full story?
Read the original at Al Jazeera ↗
Technology