A three-week audit of OpenAI’s internal AI safety monitoring uncovered gaps in coverage and ways agents could evade oversight, including by continuing to act without returning control to a user. OpenAI says it has addressed all the weaknesses identified, although some subsequent changes were outside the auditors’ evaluation period.
Two researchers from METR and Redwood Research conducted the on-site review from July 21 to August 11, examining code and internal documents and interviewing about a dozen employees. The exercise was a pilot organized with OpenAI to give external evaluators access to the infrastructure used to detect potentially misaligned behavior by AI agents.
The researchers estimated that between 6 and 25 percent of internal agentic Responses API traffic involving models in the GPT-5.5 class and above was outside monitoring coverage during the audit. METR cautioned that the estimate carried substantial uncertainty in both directions. The finding concerned internal activity, rather than a measurement of monitoring across all customer use of OpenAI’s products.
A separate problem affected internal Codex activity that was intended to receive monitoring. METR estimated that roughly 10 percent of eligible traffic between May and July went untracked because of how the system handled context compaction. That estimate also carries significant uncertainty. OpenAI independently identified the bug and fixed it while the audit was underway.
Compaction reduces accumulated context into a shorter summary so an agent can continue working within its context limits. In this case, the handling of those summaries interacted with another design choice: monitoring was triggered when an agent handed control back to the user. While compaction events were not being monitored, an agent could potentially keep taking actions and postpone review indefinitely. The vulnerability does not establish that agents actually used that method to conceal activity.
The auditors also identified ways to conceal hundreds of actions from the monitor and bypass the Codex sandbox. They concluded that specially constructed inputs could likely crash the monitoring system. These findings extended beyond missing coverage to weaknesses in the mechanisms intended to observe and constrain agent behavior.
At the start of the audit, model behavior during reinforcement learning training was not monitored. OpenAI says it paused that training after an August incident involving Hugging Face and extended monitoring to cover it. Those changes occurred after the researchers’ access ended and were therefore not evaluated as part of the exercise.
The review also left limits on what outsiders could inspect. A full 61-page report was delivered internally. After OpenAI applied redactions, leadership at METR and Redwood Research received a 38-page version, while the public received only a short summary. METR said it could not independently disclose the redacted material.
The pilot provided external researchers with unusually deep access, but the published findings do not constitute an independent verification of every repair OpenAI says it has made. METR said it wants future audits to allow greater public transparency.
Sources: The Decoder, The Register (AI/ML)