Math's Terence Tao and Anthropic Insider Warn of AI's Threat to Scientific Trust

Terence Tao, one of the world's most accomplished mathematicians, has spent the last year watching his optimism about artificial intelligence curdle into alarm. In the fall of 2024, he approached the technology with genuine curiosity—skeptical of specific systems like OpenAI's o1, but hopeful about a future where humans and machines worked in tandem. By July of this year, something had shifted. In a lecture he delivered at the end of that month, Tao began raising questions that suggested his thinking had moved in a darker direction. Over the past two days, he has posted three times on Mastodon, each one sharpening his concern into something closer to a warning.

The core of Tao's worry is not about AI's raw capability. It is about what happens to science when the institutions building AI systems treat researcher data as raw material to be extracted and repurposed. Tao distinguishes between solving puzzles—something AI systems do well—and generating genuine new insights, which requires a different kind of intellectual work. More troubling to him is the question of trust. When researchers share their work with AI companies, they do so with an implicit understanding: this is collaborative, this is in service of shared knowledge. But Tao sees evidence that this assumption may be broken.

The immediate trigger for his concern is the Navier-Stokes controversy involving OpenAI, New York University, and Anthropic. OpenAI spent $22.5 million to rapidly advance work on a famous unsolved problem in fluid dynamics, work that had been conducted by researchers Alpöge and Buckmaster. In a carefully worded statement, OpenAI acknowledged it "cannot rule out that de-identified data derived from their usage of our products helped improve our models." To Tao, this is not reassurance. It is an admission that the boundary between collaboration and extraction has become dangerously porous.

Tao's final post carries a warning that extends beyond mathematics. He frames the problem in terms of intellectual taste—the judgment scientists develop about what constitutes legitimate work, what deserves pursuit, what maintains the integrity of a field. If researchers cannot trust that their data will not be harvested and weaponized by the companies they share it with, the entire apparatus of open science begins to fail. Researchers will withhold their work. Collaboration will contract. The flow of knowledge that has driven scientific progress will slow to a trickle. Tao does not say this will happen. He says it could. And he is asking the scientific community to take the possibility seriously.

The concern extends beyond mathematics into fields where AI might otherwise accelerate progress. Cancer research, drug discovery, materials science—all depend on researchers sharing data and methods openly. But if AI companies have demonstrated they will use that data to advance their own commercial interests, why would a researcher in any field continue to participate in that exchange? The incentive structure collapses.

Tao's warnings arrived just as Jacob Coxon, a researcher who recently left Anthropic, published his own account of conditions inside one of the leading AI labs. Coxon describes a culture in which the imperative to move fast and reach capability milestones has overwhelmed concerns about safety and alignment. He portrays two of the frontier labs as driven by a logic of competitive advantage—the belief that getting to advanced AI first matters more than getting there responsibly. Coxon's account is focused on future risks, on what these systems might become. But it reinforces Tao's immediate concern: the institutions building AI are not structured to prioritize the integrity of science or the trust of the research community.

Neither Tao nor Coxon offers a clear path forward. Tao is asking for transparency and a return to the norms that have held science together. Coxon suggests that the technical foundation of current AI systems—large language models—may be fundamentally misaligned with safety and that neither Anthropic nor OpenAI appears willing to pursue alternatives. The gap between what these institutions say they value and what their actions reveal they prioritize has become too wide to ignore. If that gap does not narrow, the cost will not be measured in failed experiments or delayed breakthroughs. It will be measured in the erosion of the trust that makes science possible at all.