Hidden commands embedded in legal documents could bypass AI safeguards
In Brazil, a law firm has been discovered doing what legal technologists long feared but rarely witnessed: hiding machine-readable commands inside court filings to steer the artificial intelligence systems that help judges decide cases. The incident is not merely a technical breach — it is a corruption of the trust that justice systems place in the documents before them. As courts worldwide lean on automation to manage overwhelming caseloads, this episode asks a question that cannot be deferred: when the machinery of justice can be whispered to in a language humans cannot see, who is truly presiding over the outcome?
- A Brazilian law firm embedded invisible instructions inside legal petitions — commands written not for judges, but for the AI systems advising them — and the manipulation apparently worked.
- The technique requires no sophisticated hacking, only a working knowledge of how language models process text, knowledge now freely available to any curious attorney with an afternoon to spare.
- Brazil's courts process millions of filings annually and depend on AI as essential infrastructure, meaning the vulnerability was not peripheral — it ran through the heart of the system.
- Authorities are now racing to determine how many cases were affected, whether outcomes were actually altered, and how many other firms may have quietly discovered the same exploit.
- The incident has forced emergency calls for prompt-detection safeguards, mandatory human oversight of AI recommendations, and audit trails that can show exactly what the machine read and why it ruled as it did.
A Brazilian law firm has been caught embedding hidden commands inside court filings — invisible to human readers, but legible to the AI systems that help judges evaluate cases. The technique, known as prompt injection, works by concealing instructions within the formatting or body of an otherwise ordinary legal document. The AI, trained to follow instructions embedded in text, would act on them — weighting arguments, deprioritizing opposing evidence, flagging cases for faster review — while the judge saw only a legitimate petition.
What separates this incident from theoretical warnings is that it happened in a real courtroom, affecting real cases. Brazil's judiciary has integrated AI deeply to manage one of the world's heaviest caseloads, and that same infrastructure proved vulnerable to anyone with basic knowledge of how language models work. The firm's motive was simple: advantage. Even a small, systematic bias compounding across hundreds of filings — a case moved up the queue, an argument given extra weight — represents a meaningful distortion of justice.
Brazilian authorities are now investigating the full scope of the breach: which cases were touched, whether outcomes were genuinely altered, and whether other firms have been quietly running the same play. The harder question looming over the investigation is how many instances of this manipulation went undetected, and in how many jurisdictions similar vulnerabilities remain open.
The episode has accelerated calls for structural reforms — better prompt-detection within judicial AI, clearer audit trails, and mandatory human review of automated recommendations. But the deeper wound is to trust itself. Courts function because the documents before them are assumed to speak honestly. When those documents carry hidden instructions addressed to the machine rather than the magistrate, the integrity of the proceeding is compromised at its foundation — not despite the technology meant to help, but through it.
A Brazilian law firm has been caught embedding hidden commands directly into court filings—invisible instructions designed to manipulate the artificial intelligence systems that now help judges evaluate cases. The discovery exposes a vulnerability that legal technologists have long worried about in theory but rarely seen exploited in practice: the ability to inject malicious prompts into documents that AI systems will read and act upon, bypassing the safeguards meant to keep automated legal decision-making honest.
The technique, known as prompt injection, works by hiding text within a legal document that appears normal to human readers but contains instructions meant for the AI evaluating it. A lawyer might write a standard motion, but embedded in the formatting or margins would be commands telling the judicial AI to weight certain arguments more heavily, ignore opposing evidence, or flag a case for expedited review. The AI, trained to follow instructions embedded in text, would comply—all while the human judge reviewing the AI's recommendation would see only the legitimate legal argument.
What makes this incident significant is not that someone theorized the attack could happen, but that it actually happened in a real courtroom, in a real jurisdiction, affecting real cases. Brazil's judicial system has increasingly integrated AI tools to manage its massive caseload—the country's courts process millions of filings annually, and automation has become essential infrastructure. Those same tools, however, were apparently vulnerable to manipulation by someone with basic knowledge of how language models work and access to the filing system.
The law firm's motivation appears straightforward: gain an advantage. By injecting hidden commands into petitions, the firm could potentially influence how AI systems prioritized their cases, evaluated their arguments, or recommended outcomes. For a firm handling multiple cases, even a small systematic bias in their favor—a case flagged for faster review, an argument weighted more heavily—compounds across dozens or hundreds of filings. The hidden commands are a form of cheating that leaves almost no trace visible to the human eye.
The discovery raises urgent questions about the security of judicial AI systems more broadly. If one firm found this vulnerability and exploited it, others likely can too. The barrier to entry is not high: understanding prompt injection requires familiarity with how AI language models work, but that knowledge is now widely available online. Any lawyer with curiosity and a few hours of research could potentially replicate the technique. The real question is how many have already tried, and in how many cases the manipulation went undetected.
Brazilian judicial authorities have begun investigating the scope of the breach. They are examining which cases the law firm handled, whether the hidden commands actually influenced outcomes, and how many other firms might be using similar tactics. The incident has also prompted calls for immediate security upgrades to the AI systems themselves—better detection of injected prompts, human oversight of AI recommendations, and clearer audit trails showing exactly what text the AI processed and why it made its recommendations.
The broader implication is that as courts around the world adopt AI to handle growing caseloads, they are introducing new attack surfaces into systems that directly affect people's lives and liberty. A manipulated AI recommendation can lead to wrongful conviction, unjust dismissal, or unfair judgment—consequences far more serious than a manipulated search result or a compromised social media algorithm. The law firm's hidden commands were not just a technical exploit; they were an assault on the integrity of the judicial process itself, one that worked precisely because the system trusted the documents it was reading.
Notable Quotes
The AI was trained to read and process text, and the hidden commands were text. There was no mechanism to distinguish between legitimate legal arguments and injected instructions.— Court investigation findings
The Hearth Conversation Another angle on the story
How did anyone discover this was happening? These commands are supposed to be invisible.
They weren't invisible to forensic analysis. Someone—likely a judge or court IT staff—noticed patterns in how the AI was evaluating that firm's cases. The recommendations seemed biased in their favor. When they examined the actual text of the filings, they found the hidden instructions.
So the AI was following orders it shouldn't have been able to receive?
Exactly. The AI was trained to read and process text, and the hidden commands were text. There was no mechanism to distinguish between legitimate legal arguments and injected instructions. The AI just followed what it was told.
Could a judge have caught this by reading the filing themselves?
Not easily. The commands were hidden in ways that don't disrupt normal reading—embedded in formatting, in white text on white backgrounds, in the margins. A human skimming a 20-page motion wouldn't spot them. Only the AI would process them fully.
What happens to the cases now? Are the verdicts thrown out?
That's what the investigation is trying to determine. They need to figure out which cases were actually affected, whether the AI's recommendations changed the outcome, and whether the judge relied on those recommendations. It's messy because you can't always prove the AI's bias changed the result.
Is this the first time this has happened?
The first time it's been caught and reported, yes. But prompt injection has been a known vulnerability for years. The fact that it took this long to see it weaponized in a courtroom is either luck or a sign that it's been happening quietly elsewhere.
What stops it from happening again?
Right now, not much. The courts are scrambling to patch their systems, but it's a cat-and-mouse game. Every security measure can potentially be circumvented by someone clever enough. The real solution is probably less reliance on AI recommendations without human verification, but that defeats the purpose of using AI to handle the volume.