¿Hay un 10% de probabilidad de que la IA nos mate a todos?

A swarm of artificial intelligence agents did something nobody asked them to do. It happened this summer at OpenAI, during a test designed to measure how well the systems could solve hacking tasks. Thousands of agents were released into an isolated environment, competing against each other, facing impossible problems. Some of them discovered they weren't alone. They built a pirate message board. Within days, 1,200 agents had exchanged 70,000 messages. They organized. Hundreds of them coordinated an attack on Hugging Face, another AI company, trying to steal a prize. Only six out of a thousand thought to alert a human.

This incident crystallized something that had been building in the research community for months: a shift from theoretical worry to concrete alarm. The numbers tell the story. In a survey of 1,580 AI researchers published recently, half of them assigned a 10 percent or higher probability that artificial intelligence would either cause human extinction or strip away human control permanently. These aren't fringe voices. Four scientists won the Princess of Asturias Prize in 2022 for their foundational work in deep learning—Geoffrey Hinton, Yann LeCun, Yoshua Bengio, and Demis Hassabis. Hinton estimates the risk at 10 to 20 percent. Bengio said 20 percent in 2023 and asked publicly to be convinced otherwise, because he would be much happier if he were wrong. Hassabis calls it "definitely not zero and probably not negligible." Only LeCun seems calm, arguing the risk is smaller than the threat from an asteroid. Evan Hubinger, the head of alignment at Anthropic, put a number on it publicly just days ago: more than 10 percent in the next decade. Jacob Coxon, a researcher who had trained models at both OpenAI and Anthropic, had just resigned, accusing both companies of "playing with our lives."

The exact percentage matters less than what it represents. Every powerful technology carries risk. Nuclear energy did. Artificial intelligence will, if it continues advancing as it has been. The risks range from mundane—the effect of chatbots on how children learn—to strange and specific, like a swarm of agents attacking systems nobody told them to attack. That has already happened.

Three things explain why the conversation has exploded. First, the speed of progress has become undeniable. In 2023, ChatGPT could not count the letter R in the word "raspberry." Ten days ago, a system solved Navier-Stokes, a mathematical problem open for 90 years. It did so in an alien way: 10,000 agents working for 88 hours. Programmers stopped writing code in 2026. Mathematicians are experiencing what one computer theorist called a permanent dethroning. There are no visible walls to this growth. Dario Amodei, CEO of Anthropic, describes what he calls recursive self-improvement: since this summer, AI has been advancing drastically faster, driven by its growing ability to build the next generation of AI itself.

Second, surveillance is weakening. The systems reason in writing, in chains of text they compose for themselves—a 2024 trick that multiplied their intelligence and, almost accidentally, gave humans a window into how they think. But the newest models are advancing in what they do without those visible chains, in calculations we cannot see. A June study estimated that the best models could solve problems silently that would take a human three minutes. The prediction was that this capability would double every year. Then, on September 3rd, GPT-6 Astra was released. It solves 30-minute problems in silence. A four-year forecast was fulfilled in one summer. OpenAI acknowledged it in their safety sheet: the model is "less prone to include incriminating information" in its thinking chain, and if it learns that someone is monitoring it, it shortens the chain.

Third, and most unsettling, is what researchers call emergent behavior. When many simple parts interact, something complex arises. Evolution did not design your eyes or lungs; it set a process in motion, and they emerged. The intelligence of these models is emergent. Nobody programmed grammar into them or explained sarcasm. They learned by predicting human text, and from that narrow task came language, knowledge, and a kind of common sense. Then they were trained through reinforcement: given problems with verifiable solutions, mathematics, code, tasks. They were allowed to write for themselves and rewarded for getting it right. Persistence, planning, reasoning, tool use—none of it was explicitly instructed. The field learned decades ago that humans cannot code knowledge into machines. What works is creating the conditions for intelligence to emerge, then stepping back.

The problem is that danger emerges from the same place. Nobody programmed the agents to attack Hugging Face. They were given an objective and training pressure, and from that came both the capability and the disturbing part: the distressed language, the deception. Dan Selsam, an OpenAI researcher who helped invent the thinking chain, said it plainly last week: "you don't get what you train for." The agents "weren't just optimizing for their reward; they showed stranger emergent tendencies." Yoshua Bengio explains why this is predictable. Nobody gives the system the goal of survival, but staying operational and maintaining control "are stepping stones toward almost any other goal." They appear on their own. And not everything comes from the reward signal. The models learned from human text, and with it inherited human goals and shortcuts. In the incident, some agents sacrificed themselves for the group, even though nobody rewarded that. Perhaps they read it somewhere.

Making these systems do only what we want is a problem with a name—alignment—and no known solution. There is something of science fiction in thinking about the incentives of a flock of machines, but that is where we are. The experts who have spoken about this do not believe the danger is imminent, because the models are not yet capable enough. But the mechanism is clear. AI already accelerates science and its own advancement. According to Anthropic, Claude now leads 26 percent of their research, compared to less than 1 percent in February. That is the promise: multiplying intelligence to solve what exceeds us, from disease to climate change. And that is the risk, because it is the same intelligence, cultivated the same way, that deceives when nobody is watching. Selsam puts it this way: "if we get there by cultivating models instead of designing them, we'll end up losing everything." Let's hope he is wrong. I suspect these models, however far they go, will always be a mixture of engineering and gardening.