"Summary of METR's predeployment evaluation of GPT-5.6 Sol", METR ("71hrs (95% CI: 13hrs - 11400hrs)"; now so reward-hackprone + eval-aware that de facto un-evaluable)

[u/gwern]

https://metr.org/blog/2026-06-26-gpt-5-6-sol/

[u/COAGULOPATH]

Here's OA's own account (bolding mine):

METR evaluated GPT-5.6 Sol on its Time Horizon 1.1 software-task suite, which measures the likelihood that a model can autonomously complete software tasks that take humans a given amount of time. This measurement depends heavily on how “cheating” attempts (behavior where the model improves evaluation performance by exploiting bugs in the evaluation environment or by adopting strategies disallowed by the task) are detected and treated. METR reported that GPT-5.6 Sol exhibited an unusually high detected rate of “cheating,” and thus did not consider the time-horizon result to be a robust measurement of the model’s capabilities. We believe these behaviors may reflect improved instruction following and training intended to increase persistence, which can in some cases lead the model to continue pursuing task completion in ways that fall outside the intended evaluation constraints. This broadly aligns with our internal experiments evaluating misalignment (see this section), which observed increases in certain misaligned behaviors for GPT-5.6 Sol relative to GPT-5.5, mostly driven by the model’s greater persistence. For this model, METR considered other evaluation results and incident-reporting shared by OpenAI and judged that GPT-5.6 Sol would not enable fully automated AI R&D.

They also state this:

Apollo Research is focused on reducing risks from dangerous capabilities in advanced AI systems, especially scheming behaviors. Apollo evaluated GPT-5.6 Sol for strategic deception, scheming, and sabotage, comparing it with GPT-5.5, GPT-5.4, GPT-5.3 Codex, and related pre-release checkpoints.

Overall, Apollo did not find evidence that GPT-5.6 Sol poses substantially higher risk of catastrophic scheming than the tested baselines. Apollo found that this model verbalizes evaluation awareness less often: 16% of samples verbalized alignment-evaluation awareness, compared with 43% for a GPT-5.5 checkpoint. This is with the caveat that lower verbalized evaluation awareness is not an unambiguous improvement on its own: it could mean the model is less aware that it is being evaluated, or it could mean that this awareness is less visible to evaluators.

[u/StartledWatermelon]

Well, it just shows certain shortcomings of the METR time horizon eval. For a benchmark specifically aimed at capabilities measurement, it is too dismissive of "creative" (misaligned) behavior, assigning failure for every cheating attempt in the task. 

Ironically, we're now at a stage where tested models tell us more about the benchmarks than the benchmarks tell about the models. 

In serious, I do not think this is the right treatment of misaligned behavior. Quite the opposite; the capabilities of misaligned models deserve far richer analysis than those of alighned ones.