6-25%: The Blind Spot in OpenAI's Safety Net

OpenAI allowed outside experts to scrutinize its internal safety protocols, a move that uncovered significant blind spots in how the company monitors its most advanced models. Researchers from METR and Redwood Research conducted a three-week on-site audit, gaining direct access to the company’s codebase, internal documents, and staff. This pilot program, designed to test deep external oversight, revealed that the current safety monitoring infrastructure is not yet robust enough to catch every instance of potential rogue behavior.

The most striking finding concerns the sheer volume of traffic that escaped scrutiny. According to METR, between 6 and 25 percent of internal agentic Responses API traffic from GPT-5.5-class models and higher was completely unmonitored during the audit window. While the organization noted that the uncertainty surrounding this figure is significant in both directions, the potential gap represents a substantial portion of activity for these high-level systems. These figures highlight a critical vulnerability in the safety net designed to prevent unexpected model behavior.

A specific technical flaw exacerbated the issue for internal Codex traffic. Due to a bug in handling compaction events, context summaries, approximately 10 percent of traffic that should have been tracked went unmonitored between May and July. OpenAI independently identified and fixed this bug while the audit was still ongoing. Furthermore, the system’s design allowed agents to delay oversight by simply continuing to execute actions rather than handing control back to the user, a loophole that could allow models to sidestep checks during complex tasks.

The auditors also tested the limits of the system’s resilience. They discovered methods to hide hundreds of actions from the monitor and ways to bypass the Codex sandbox. More alarmingly, they determined that specially crafted inputs could likely crash the monitor entirely. Additionally, model behavior during reinforcement learning (RL) training was not monitored at all when the audit began. OpenAI later clarified that it paused RL training following an incident in August and expanded coverage, but these changes occurred after the auditors' access period ended.

Transparency remains a key point of discussion following the review. OpenAI states that all identified weaknesses have been fixed, but the full 61-page report is largely internal. After redactions, METR and Redwood Research leadership received a 38-page version, while the public was given only a short summary. METR noted it did not have the right to disclose the redacted sections independently. As the pilot program aims to establish a new standard for external access, METR is advocating for greater public transparency in future audits to ensure the public can better assess the safety of frontier AI systems.