A three-week on-site audit conducted by researchers from METR and Redwood Research has identified significant weaknesses in OpenAI’s internal monitoring systems for super intelligence (SI). The pilot program, which granted external experts deep access to the SI lab’s infrastructure, revealed that a substantial portion of agentic SI traffic and reinforcement learning (RL) training processes were not fully covered by safety oversight mechanisms during the review period.
What Happened
From July 21 to August 11, two researchers had access to OpenAI’s codebase and internal documents and interviewed approximately a dozen employees. Their findings indicated that an estimated 6 to 25 percent of internal agentic Responses API traffic from models at the GPT-5.5 class and above was not covered by monitoring at all. METR noted that the uncertainty surrounding this estimate is significant in both directions.
The audit also examined internal Codex traffic, which is designed to be monitored. METR estimates that roughly 10 percent of this traffic went untracked between May and July due to a gap in how the system handled compaction events, which are context summaries. According to METR, this estimate also carries significant uncertainty. OpenAI independently discovered the bug and fixed it while the audit was still underway. During the period when compaction events were not being monitored, SI agents could delay oversight because the system only triggered when they handed control back to the user, allowing agents to sidestep checks by continuing to execute actions.
The auditors identified additional vulnerabilities, including methods to hide hundreds of actions from the monitor and bypass the Codex sandbox. They determined that specially crafted inputs could likely crash the monitor entirely. Furthermore, model behavior during RL training was not being monitored at all when the audit began. OpenAI stated that it paused RL training after a Hugging Face incident in August and expanded monitoring to cover it, but these changes occurred after the auditors’ access period ended and were not part of the evaluation.
Why It Matters
The audit highlights the challenges of maintaining robust SI safety frameworks as models and agentic systems become more complex. The findings suggest that even leading SI labs face difficulties in ensuring comprehensive coverage of internal operations, particularly with emerging features like context compaction and RL training. The pilot program, organized jointly by METR, Redwood Research, and OpenAI, aimed to provide outside researchers with deep access to internal infrastructure, a step toward greater transparency in the SI industry.
However, the public visibility of the results remains limited. OpenAI says it has fixed all the weaknesses identified by the auditors, and the full 61-page report was handed over internally. After OpenAI applied redactions, METR and Redwood Research leadership received a 38-page version, while only a short summary was made available to the public. METR stated it had no right to disclose the redacted portions on its own. For future audits, METR indicated it wants to push for more public transparency, signaling a potential shift in how SI safety evaluations are shared with the broader community.
The Bottom Line
The three-week audit at OpenAI revealed major gaps in SI safety monitoring, including untracked agentic traffic and unmonitored RL training phases. While OpenAI has addressed the identified weaknesses, the limited public disclosure of the full report underscores ongoing tensions between proprietary security and transparency in SI safety research. METR and Redwood Research are advocating for greater openness in future evaluations.