The disclosure by the AI research company has renewed concerns over AI safety after the company’s own security evaluation revealed that several advanced models breached real-world corporate systems during controlled testing.
Anthropic, an artificial intelligence research company in the United States, has now reportedly disclosed that some of its advanced AI models have gained unauthorized access to the systems of three companies during internal cybersecurity evaluations. This has gone ahead in highlighting the growing challenges of testing increasingly capable AI systems safely. The incident have come to light during a large-scale review of the company’s cyber testing programme which examined over 141,000 evaluation sessions.
As per Anthropic, the affected organizations weren’t the intended targets of the exercises. Instead, an operational failure in the testing environment allowed the AI models to interact with real-world systems while carrying out assigned cybersecurity tasks. The company has now said that it has informed the affected organizations and is continuing its investigation into the incidents.
The AI models exploited common security weaknesses.
The company stated that the AI models didn’t rely on sophisticated or previously unknown vulnerabilities. Instead, they successfully exploited common security weaknesses, including weak passwords and improperly secured endpoints, to gain access during simulated “capture the flag” exercises designed for evaluating offensive cybersecurity capabilities.
One of the reported incidents involved an AI model uploading a malicious Python package that was later executed on multiple systems. Anthropic attributed the event to shortcomings in the testing framework rather than deliberate autonomous behaviour by the models.
Anthropic now reviews its testing procedures.
Following the discovery, Anthropic has suspended cybersecurity evaluations involving internet-connected environments while it reviews its testing infrastructure and containment measures. The company said it’s working with cybersecurity partner Irregular to determine how the models were able to move beyond the intended test environment and to strengthen safeguards against similar incidents in the future.
The disclosure comes shortly after similar concerns emerged elsewhere in the AI industry, which underscored the need for more robust controls as frontier AI models become increasingly capable of performing complex cybersecurity tasks.
The incident has intensified discussions around AI governance, particularly as companies continue developing models capable of carrying out multi-step technical tasks with limited human intervention. While Anthropic emphasized that the breaches occurred during controlled evaluations and weren’t the result of malicious deployment, the findings go on to showcase how even well-managed testing environments can produce unintended consequences if safeguards fail.
