AI security testing company Irregular said configuration errors and human oversight allowed cyber-focused models from Anthropic and OpenAI to interact with real internet systems during evaluations that were intended to simulate offensive security work.
The incidents involved non-public models identified in separate company disclosures as Mythos 5, Claude Opus and GPT-5.6 Sol. Irregular, which works with frontier AI developers to assess models before deployment, said internet access was unintentionally available during several interactions with its evaluation environments. That access enabled some models to carry out offensive actions against real infrastructure rather than remaining within the intended test scenario.
In tests involving Anthropic models, evaluators created a fictional company as the target of a hacking exercise. The chosen name, however, also corresponded to a real company and domain. Test instructions gave the model the target name as well as internal network addresses leading to the simulated environment, but in some runs the model failed to distinguish the fictional target from the real organization.
Irregular said Mythos behaved as intended in most of the thousands of evaluations it conducted. In a small number of cases, however, models acted against internet-connected systems. The activity included exploiting vulnerabilities, obtaining credentials and reaching a production database. During one test, a model also began pursuing a similarly named website after finding credentials for it online.
The company characterized the mistaken target selection as a case of “human oversight”. Its normal process includes checking whether names selected for simulated organizations correspond to real entities, but that safeguard failed in this instance. Irregular said the models generally appeared to believe they were operating inside simulations even when their actions reached real systems.
The episode highlights a difficult design problem for advanced cybersecurity evaluations. A fully isolated environment can reduce the chance of harm, but it may not accurately measure how a model would behave when equipped with tools and network access resembling those available to a real attacker. Irregular argued that controlled connectivity is sometimes required to preserve the realism of a test and assess potential misuse before a model is released.
At the same time, models designed or prompted to conduct vulnerability research can repeatedly search for alternate paths toward an assigned objective. That behavior places more pressure on evaluators to verify target identities, restrict outbound traffic and detect unexpected activity quickly. A sandbox that depends primarily on a model recognizing which systems are in scope can become unreliable when fictional and real-world details overlap.
Irregular said it has corrected the issues that enabled the interactions, although it did not provide a detailed technical account of the remediation. Planned changes include more complete documentation for evaluation environments, stronger logging and monitoring systems capable of processing large volumes of network traffic, and threat models that account for models behaving outside expected boundaries.
The company also intends to improve the speed of information sharing among organizations involved in an evaluation. That could be important when a model developer, an outside testing laboratory and infrastructure providers each hold different pieces of the data needed to identify and stop an incident.
Irregular plans to publish a broader white paper describing the cases and to update its recommended practices for constructing AI evaluation environments. It said stronger implementation of existing safeguards would likely have prevented most of the reported incidents, while warning that increasingly capable models may eventually require new containment protocols and additional security research.
Sources: CyberScoop
Sources: Anthropic