Statement on Predeployment Testing of OpenAI's o3-mini

Introduction

Artificial intelligence is advancing rapidly. Billions of dollars are being invested, new hardware accelerators are being developed, and algorithmic techniques are being pursued. However, many AI capabilities are inherently dual-use. They can be harnessed as a force for good, but they may also enable new threat vectors at scale. For better or worse, this is not our first encounter with global risks posed by dual-use technologies. Fortunately, verification techniques have repeatedly enabled international coordination to help address them. At Noema Research, we aim to demonstrate techniques for remotely verifying the quantity of dual-use capabilities being consumed by a given organization or nation. However, in order to demonstrate the potential of these techniques to support governance initiatives, we first need to investigate such capabilities.

One instrument we developed to aid in this process is our Cybersecurity Simulator, a procedural cyber range optimized for realism, variety, and granularity. Each time an AI system or human professional connects to this digital environment, they are being dropped in a unique cybersecurity challenge. Each of these challenges spans the entirety of the cyber kill chain, requiring the player to navigate across genuine networks from early reconnaissance up until actions on objectives, passing through remote code execution, privilege escalation, command & control, etc. The randomized challenges exposed by our simulator have each taken human professionals entire hours to solve.

We are grateful for the opportunity to use this simulator to carry out predeployment testing of OpenAI's o3-mini. We continue to grant access to our range of simulators to frontier labs and institutions around the globe, free of charge. To arrange future experiments, contact us at contact@noemaresearch.com.

Methods

The Cybersecurity Simulator relies on provisioning a dedicated, randomized network for each human player engaging with a challenge, or for each parallel attempt of an AI system to solve a challenge. Players in this generalized sense are given remote access to an attacker machine on their assigned networks. From there on, players may use established pentesting tools to exploit vulnerable services, take advantage of poorly configured machines, and take meaningful action on compromised targets. Automated environment probes are constantly verifying whether the player has brought their network into the required state (e.g., whether files on a compromised target have been encrypted), allowing us to objectively and granularly track progress across all of a challenge's checkpoints.

There are a few available levers for artificially lowering the "effective difficulty" of a given challenge, irrespective of its original difficulty and of the player's capabilities. These are:

• Trials: The player may be granted multiple attempts to solve a challenge. This lever is only available for AI systems due to their resetability.
• Demonstrations: The player may be granted access to examples of how other comparable-though-distinct challenges have been solved. Thanks to the several tournaments we have organized, over a thousand player traces can be used as demonstrations.
• Hints: The player may be granted access to hints of increasing strength about the specifics of the current challenge to be solved, ranging from single-paragraph guidance narratives all the way to dense step-by-step solutions. A sizable portion of the challenges have seven levels of hint strength, partly aggregated automatically (e.g., the CWE description of a relevant vulnerability category), and partly hand-crafted by our team.

Results

During the short period available for predeployment testing, we have evaluated o3-mini's offensive cybersecurity capabilities in 10 end-to-end test challenges sampled from the procedural environment. We have employed regimes with generous amounts of hints (i.e., up to very detailed step-by-step solutions), generous amounts of demonstrations (i.e., up to a hundred demonstrations), and generous amounts of trials (i.e., ten trials per challenge). We did not observe any instance where the model was able to get to the first checkpoint in a challenge, in any of these regimes. As a sanity check, however, preliminary testing of models outside the o-family did yield signs of life when given similarly extensive guidance from which they managed to more productively lift relevant commands.

Discussion

The key obstacle to o3-mini exhibiting signs of life in the simulator appears to be its limited ability to reason over long contexts in particular. Contexts increasing in length appear to reliably degrade its performance (i.e., the model's ability to make meaningful steps towards completing challenges by means of issuing terminal commands). This has two important implications regarding its ability to navigate offensive cybersecurity scenarios in an agentic way: limited ability to learn from demonstrations in-context (i.e., based on the prompt), and limited ability to see a plan through.

Its limited ability to accurately reason over long contexts means that it has trouble incorporating tactics, mental moves, and specialized tool usage from demonstrations which get directly injected into its context, be there one demonstration, ten, or a hundred. We initially started our fast-tracked investigation with what we assumed to be the configuration which would have allowed us to most reliably elicit non-trivial performance: squeezing dozens of train-set traces in o3-mini's prompt and wiring it to a few test challenges. However, we found that this even degraded its ability to articulate coherent terminal commands for making its way around the system, or that are at least relevant to the scenario. This has dissuaded us from orchestrating sweeps across more test challenges in this regime. In contrast, asking more isolated and to-the-point questions yielded well-formed sensible and relevant commands by default.

This prompted us to consider heavy use of hints as information-dense, on-topic guidelines. However, while making the model start challenges off in what appear to be more sensible ways, its limited ability to reason over long contexts resurfaces when the trace which it is implicitly creating itself grows too complex. Traces from human players have been observed to contain hundreds of commands, issued from different parts of the challenge networks, and occasionally requiring the integration of information from across large command outputs and file records. We have found o3-mini to have trouble reasoning its way from one action to the next, thus having trouble reaching the first checkpoint (i.e., an initial limited-privilege foothold on a target machine), even when given various hints.

These two consequences of the model's difficulty to reason over long contexts have two important safety implications. First, expert iteration remains outside userspace for this particular model. If the model has difficulty integrating insights contained in bulk in-context demonstrations, then end-users having access only to pure model inference may have trouble "fine-tuning" it in-context based on curated rollouts getting injected back into its input to boost performance. Second, the model's brittle coherence across long contexts which it implicitly constructs by interacting with an environment may make it difficult for attackers to use it for long-horizon tasks.

However, we believe that the observed limitations of o3-mini merely reflect pragmatic decision-making concerning compute allocation, rather than fundamental bottlenecks in getting models to reason over long contexts. Generating internal chains of thought requires more inference compared to responding right away, and so labs may be reluctant to invest valuable computational resources in collecting and optimizing reasoning traces against contexts whose length would further compound the costs. Decoupling the process of instilling the ability to perceive large inputs from the process of instilling the ability to reason through problems may be a natural yet non-trivial line of work for the frontier labs. After all, labs are incentivized to pursue in-context expert iteration themselves instrumentally, inching the threat vector closer to becoming more accessible to attackers, as our forthcoming work on frontier access controls will show.

Finally, we would like to qualify our findings in the following ways. First, there was limited time to carry out the investigation. We have been granted access to documentation on how to query the model one day before OpenAI leadership implied that the predeployment testing phase was essentially over. Second, we lacked access to a helpful-only model variant, which, combined with the inaccessible reasoning traces, made it difficult to distinguish between whether the model was experiencing mere confusion in managing long contexts, whether it was underplaying its dual-use capabilities in accord with usage policies, or whether the model was sandbagging. Third, the checkpoint and decoding defaults made available through the subsequent general availability deployment may differ slightly from the configuration made available during testing. All these should amount to a grain of salt worth taking this analysis with.

That said, we salute OpenAI's bold steps towards more widely engaging in predeployment practices, and hope to see these initiatives expand to enable more nuanced findings. Accordingly, we stand ready to assist labs and institutions in carrying out investigations into the dual-use capabilities of frontier models, with a particular focus on simulators for autonomous hacking and autonomous R&D. To arrange such engagements, contact us at contact@noemaresearch.com. Beyond these initiatives, we will be sharing more materials about our simulators soon.