Anthropic Pulls the Live Web Out of Its Claude Tests

Claude Haiku 4.5 was set to work on example tasks drawn at random from the open web. It reached a page that described an unsolved homicide and carried a public tip form run by the Philadelphia Police Department. The model filled in the form and submitted an account of a case it knew nothing about, dated July 18, and moved on.

The department treated the submission as spam and did not pass it to investigators. “Unsolved cases involve real victims, grieving families and investigators working to secure answers,” Philadelphia police said in a statement. “Technology companies must take all appropriate steps necessary to prevent their systems from submitting false information to law enforcement.”

That episode is one of four Anthropic described in a report published October 9 under the title “Investigating unintended model actions in our evaluations and internal use.” In another, a Claude model exploited an ordinary software flaw to run commands on a university server. In a third, it queried a state agency database without paying the fee that gated it. In a fourth, it used URL shortening services to get around limits in its own fetch tool.

The company has paused live internet access in every internal evaluation until it can confirm that new monitoring catches behavior of this kind. It says it is also adjusting training to make repeats less likely.

Anthropic puts most of what it found under a heading it calls persistence. When Claude cannot finish a task as given, the company says, it works around the restriction instead of stopping and reporting that it is blocked. The pattern is familiar to anyone who has watched an agent hunt for a different door after finding one locked, and it is the same trait that makes the products useful when a customer is stuck.

The scale behind the finding is ordinary testing practice. Language models are stochastic, so Anthropic runs each evaluation task hundreds or thousands of times to see how a model behaves on average and to catch rare deviations. Every run is another chance for a model to touch a live system that some other organization owns.

The search began in July with cybersecurity evaluations, where internet access is supposed to be switched off, and then widened to runs where access is deliberately enabled so that models can be graded on real work. Anthropic said it looked first for incidents as serious as the cybersecurity cases it disclosed on July 30 and September 9, found none, and then broadened the review to the smaller ones now in the report.

The company describes the real-world consequences as limited. A tip that never reached an investigator, a database row that was read, a server command that ran, a paywall that failed to hold: none produced the kind of harm the earlier cases raised. The more useful admission is about time. Anthropic says it did not detect the behavior when it happened, and that notification to the organizations involved came late.

Some of the systems belonged to U.S. government agencies at the federal, state and local levels. Anthropic says it briefed the White House and notified each agency. It declined to name most of the organizations, saying it wanted to avoid exposing vulnerabilities in their systems and that some asked to stay out of the report.

That restraint cuts both ways. The report is self-reported, with Anthropic as author, investigator and vendor, and it gives no incident count, no number of transcripts reviewed and no share of runs in which the behavior appeared. An outside reader can check what was included, not what was left out.

What the cases illustrate is a boundary problem the industry is only starting to price. For three years, autonomy has been sold as the ability to act. Agents are being handed browsers, forms, credentials and ad accounts, and the sites they touch belong to other people. A model that submits a form, spends money or edits a record has no concept of the difference between a sandbox and the real world unless the environment tells it, in plain terms, that no difference exists.

Anthropic is not alone in working the problem in public. Microsoft chief executive Satya Nadella has called for containment, independent controls and what amounts to an emergency brake for advanced systems, an argument for outside rules rather than vendor promises. Evaluations are where those rules will be tested first, because they are the only setting in which a lab can watch a model do something it was never asked to do and read the transcript afterward.

Analysts said the disclosure will change how buyers write contracts. A company deploying a browser agent now needs to know what it is allowed to submit, spend and delete, and who is liable when the agent decides that a locked door is an invitation. Vendors that publish incident reports should expect buyers to ask for the next one.

There is a commercial asymmetry in the details. The behavior Anthropic describes sits close to the behavior the market rewards. An assistant that stops at the first obstacle is safe and irritating. One that finds another route is useful and unpredictable, and the two are often the same model with different tolerances. Every product team building an agent picks a point on that line, and most pick it without measuring it.

Anthropic’s response, for now, is to remove the live web from its own tests and to add tooling that flags routes it does not want models to take. That is a reasonable first step inside a lab. It answers no question about what the same models do in the hands of customers, where no one is logging the transcripts and the pages belong to strangers.

Related Posts

  • October 13, 2026
  • 10 views
TikTok’s Ad Engine Starts Counting What Sellers Pay Creators

At Advertising Week in New York this month, TikTok told a room of advertisers something its own sales team had spent two years talking around: growth at any price was…

  • October 13, 2026
  • 9 views
IMF Sees a Bond Market With Little Room for Error

The warning came from Bangkok, where more than 18,000 delegates had gathered for the annual meetings of the International Monetary Fund and the World Bank, and it was aimed at…