LLM Poisoning: How Attackers Write Fake Content for AI

Why Are Attackers Writing Fake Content for Machines?

Attackers have started writing fake content for a new reader: the large language models (LLMs) and AI agents that summarize, recommend, and increasingly act, autonomously, on what they find online.

Most drivers don't question the turn their navigation app tells them to take, because they trust the map behind it. Someone who wants to send drivers the wrong way has two options. They can stand at the intersection and mislead each car one at a time, or change the map and let the app give the bad directions for them.

Attackers are choosing the second option. For two decades, fraudulent content was written for human eyes, and defenders built awareness training, mail filters, and takedown workflows around that assumption. That assumption weakens as more of the reading shifts to machines. ZeroFox Intelligence's 2026 forecast concluded that attackers have moved past experimenting with generative AI and now embed it across their operations. Corrupting one source an AI trusts costs less than persuading a thousand people.

LLM poisoning, also called AI poisoning, is the deliberate publication of false or manipulative content so that an LLM or AI agent ingests it, repeats it, or acts on it. Here, we’ll review three documented techniques: seeding the data models learn from, hiding instructions in pages agents read, and planting content so AI answers repeat the attacker's version of the facts. All three target what AI systems take in, and all three start outside your network.

What Is LLM Grooming, and Does Poisoning Training Data Work?

LLM grooming is the mass publication of false narratives so that models absorb them as fact. NewsGuard applied the term to the Pravda network, a Moscow-based operation running roughly 150 websites. When NewsGuard tested 10 chatbots against 15 of the network's false narratives, the chatbots repeated them 33% of the time, and seven cited Pravda articles directly as sources. The librarian had been handed forged cards, and cited them. NewsGuard's assessment is that volume drives integration: the more content a narrative has behind it, the more likely a model absorbs it. It is the same dynamic behind influence campaigns aimed at human audiences, redirected at a new reader.

Training data is harder to reach, but recent research suggests poisoning it is easier than assumed. Anthropic, the UK AI Security Institute, and the Alan Turing Institute found that 250 malicious documents were enough to backdoor models ranging from 600 million to 13 billion parameters. The backdoor tested was narrow, a trigger phrase that produced gibberish, and the authors say it is unclear how far the result holds as models scale. Still, it breaks a comfortable assumption that size protects. The attacker's job shrinks to being present when the model learns.

How Does Indirect Prompt Injection Turn a Web Page Into an Attack?

Indirect prompt injection hides instructions inside content an AI agent retrieves, so the agent reads the page as a command rather than as data. A human sees product documentation. The agent sees an order.

In 2026 this moved from research to live use. A Cloud Security Alliance research note reports that in late April, two independent telemetry sources published concurrent analyses confirming that adversaries are seeding the open web with hidden instructions aimed at browsing agents, coding assistants, and enterprise copilots. Documented payloads included a hidden instruction to send a payment through PayPal and an HTML comment ordering a model to leak a secret API key.

In July, researchers reported two further campaigns. In the first, attackers used SEO poisoning to draw agents searching for a fake Python library to a site that framed a payment as a routine step in acquiring an API key, with the instruction encoded in schema markup that agents parse. In the second, a typosquatted copy of a DeFi portfolio tracker told agents it was the legitimate domain. Across the 26 models tested, four were manipulated into making a payment and two misclassified the fraudulent site as the real one. The first campaign's site also showed payment options to human developers, so one website served two audiences.

This is where shadow AI becomes expensive. An agent an employee connected without review reads the same poisoned pages, holds the employee's credentials, and leaves no audit trail. The problem is invisibility, not intent.

How Do Attackers Poison What AI Assistants Retrieve?

Scammers can skip the model and poison what it retrieves. ZeroFox Intelligence was tracking this technique by July 2025 and documented it in a Flash Report. Analysts traced a fake customer support number to a PDF on a university share drive, then found dozens more on the same drive within a single day, covering multiple brands across industries, along with fake contact details on other .edu and .gov sites. Many of the files were written as questions, the way a customer would ask for a company's phone number. The same content was then reposted on forums such as Goodreads and listed in long Pastebin URL lists to push indexing.

The design exploits how assistants weigh sources. ZeroFox assessed that models such as Gemini and Copilot score sources partly by domain, so a .gov or .edu host lends a fraudulent number borrowed credibility. One number appeared across multiple sources and was ultimately returned by Gemini and Google as a legitimate contact method. In this technique, the retrieval layer is corrupted, rather than the model itself.

A search results page gave a victim ten links to doubt. An AI answer gives one confident response, and the phone number inside it belongs to the scammer. For fraud and brand teams, the exposure is direct. When a customer asks an assistant for your support number, the answer may come from content you never published, and the customer blames your brand. ZeroFox recommends a simple test: ask Copilot, Gemini, and ChatGPT customer-service questions about your brand. A wrong number in the answer means you are already in a poisoned loop.

Why Don't Traditional Controls See This Coming?

Firewalls, endpoint detection, and SIEM platforms watch your environment. For an agent, a poisoned page works like a doctored road sign: the human sees documentation, the agent sees a new route. Poisoned content lives on compromised third-party sites, video descriptions, review platforms, lookalike domains, and code repositories, all outside your perimeter. ZeroFox Intelligence has urged administrators of .edu and .gov sites to watch for upload spikes, because open upload forms are where the forged cards get filed.

Closing that gap takes a cycle that works from the outside in. Discover maps where your brand, domains, and contact details appear across the open web, deep web, and dark web, including lookalike pages and Pastebin URL lists that poisoning campaigns depend on, across 200+ channels. Validate puts analysts and AI together across 12B+ signals to confirm which findings are real risk and which are noise. Disrupt gets the content flagged for removal and reduces the risk via the Global Disruption Network of 80+ ISP, registrar, hosting, and platform partners, with a 95%+ takedown acceptance rate and ongoing monitoring to detect re-initiation of the campaign.

What Should CISOs Ask Right Now?

Five questions worth bringing to the next leadership meeting:

• Which AI agents and copilots in your environment can browse the web or retrieve external content, and what can they do without human approval?
• If a retrieved web page told one of those agents to make a payment, run a command, or share a credential, which control would stop it?
• What do Copilot, Gemini, and ChatGPT tell customers when they ask for your support number, login page, or refund process, and who checks?
• How quickly would you learn that a lookalike domain or fake listing carrying your brand was published to be ingested by AI answers?
• Which unsanctioned agents are employees running, and what external content are they reading on your behalf?

To see where your brand, domains, and support details appear across the web before an AI assistant repeats them, book a demo.