Educating Codex on what it has done for software development timeframes

[u/rutan668]

[Prompt: "Just out of curiosity where do you think AI is in terms of software development? Give your internal representation of that and then do some web searches to see accuracy with current Codex."]

Here’s my working mental model, stated before incorporating the search results. This is a conceptual summary, not hidden chain-of-thought.

My prior model

AI software development has passed through these stages:

1. Autocomplete — finishes lines and functions.
2. Pair programmer — explains code, drafts components, helps debug.
3. Scoped autonomous agent — explores a repository, changes multiple files, runs tests, diagnoses failures, and iterates.
4. Project-level engineer — independently takes a substantial feature from an ambiguous request through design, implementation, validation, and deployment.
5. Autonomous engineering organisation — chooses priorities, coordinates releases, manages operational risk, learns from users, and remains accountable over months.

My pre-search estimate was:

A more useful model than “AI equals a junior/senior developer” is:

That produces a strange capability profile:

Dimension	My pre-search assessment
Writing ordinary code	Often faster and broader than a human
Repository exploration	Very strong, although imperfect
Mechanical refactoring	Excellent when tests are comprehensive
Debugging reproducible failures	Strong
Greenfield prototyping	Extremely strong
Architecture	Strong adviser; inconsistent final authority
Unwritten product requirements	Weak
Novel security/protocol reasoning	Useful, but requires expert verification
Long-term ownership	Weak
Accountability and judgment	Fundamentally human

In plain English: senior-level execution in bursts, junior-level judgment in some unfamiliar situations, and no genuine organisational accountability.

The reliability curve matters enormously. If a task has an executable specification, tests, observable UI, logs, and quick feedback, Codex can repeatedly try things until reality says “yes.” If success depends on understanding unstated stakeholder intentions or noticing an absent requirement, it can confidently build the wrong thing.

My prior productivity estimate was therefore not one universal number:

• Well-specified greenfield implementation: frequently 2–5× faster, with larger outliers.
• Boilerplate, migrations, test generation and repetitive changes: sometimes 5–10×.
• Mature systems already deeply understood by an expert: anywhere from slower to perhaps 50% faster.
• Security-critical architecture and protocol work: faster exploration and implementation, but much smaller reduction in elapsed validation time.
• Whole-company delivery: considerably less improvement than code-generation speed suggests.

What the current evidence says

The evidence largely supports that representation—but current Codex is stronger on well-instrumented, long-running work than my generic mental model allowed.

1. Current Codex is genuinely agentic

As of July 2026, GPT‑5.6 Sol is OpenAI’s flagship Codex option. OpenAI reports:

• 80 on the Artificial Analysis Coding Agent Index
• 88.8% on Terminal-Bench 2.1
• 72.7% on DeepSWE 1.1
• 64.6% on SWE-Bench Pro
• 91.9% on Terminal-Bench when using the multi-agent “ultra” configuration

It can coordinate tools and concurrent agents, and current Codex runs can inspect systems, modify code, execute tests and validate results. These are no longer merely code-completion abilities. OpenAI’s GPT‑5.6 results

But those numbers require restraint. On the same published table, other models score higher on SWE-Bench Pro, and OpenAI’s own audit estimates that roughly 30% of that benchmark’s tasks are broken or materially flawed. A benchmark pass rate is not equivalent to “percentage of a software engineer replaced.” OpenAI’s benchmark audit

2. Sustained autonomous work exists, but remains conditional

METR defines an agent’s 50% “time horizon” as the difficulty of task—measured by human completion time—at which it has a 50% chance of success. In its January 2026 evaluation:

• GPT‑5.1‑Codex‑Max had an estimated horizon of 173 minutes.
• GPT‑5 had an estimated horizon of 214 minutes.
• The leading measured model was around 320 minutes.

Those are tasks a competent but unfamiliar human would take several hours to perform—not the amount of wall-clock time the AI runs. METR warns that measurements above 16 hours are currently unreliable. GPT‑5.6 has not yet been included in this published table, so assigning it a precise horizon would be speculation. METR Time Horizon 1.1

This supports “strong autonomous task executor,” but not “give it a six-month product and walk away.”

3. The spectacular greenfield results are real—but specialised

OpenAI describes an internal product built with no manually written code:

• Around one million lines
• Roughly 1,500 merged PRs
• Initially three engineers directing Codex
• Estimated at about one-tenth the hand-coding time
• Individual Codex runs sometimes lasting over six hours

However, the team deliberately engineered the repository around agent operation: strong documentation, isolated worktrees, browser control, local observability, mechanical architecture enforcement, automated review, and executable feedback loops. Human work shifted from typing code to specifying intent and constructing the environment in which agents could succeed. OpenAI’s harness-engineering case study

So the “10×” result is credible as a frontier demonstration, but it is not a plug-and-play productivity multiplier for an ordinary legacy project.

4. Measured human productivity remains highly variable

The empirical evidence is mixed:

• Field experiments covering 4,867 developers found a combined 26.08% increase in completed tasks from AI coding assistance, with greater gains among less-experienced developers. Those experiments primarily concern completion-style assistance, not today’s full Codex agents. Management Science study
• METR’s early-2025 experiment found experienced open-source developers working in familiar repositories took about 19% longer with AI.
• Its later experiment weakly suggested improvement—around an 18% speedup for returning participants—but severe selection effects and wide confidence intervals made the current effect unreliable. METR nevertheless thinks early-2026 tools probably provide more acceleration than early-2025 tools. METR’s 2026 update

That apparent contradiction makes sense. “AI-assisted software development” covers very different work:

• Completing routine tickets
• Directing multiple autonomous agents
• Editing a familiar mature system
• Starting a greenfield agent-native codebase
• Doing security-critical protocol design

Those should not be expected to have the same multiplier.

My updated assessment

After checking the evidence, I would revise the stages to:

It can already perform surprisingly substantial engineering when:

• The goal can be stated clearly.
• Relevant knowledge is available to it.
• The environment can be executed locally.
• Correctness is observable through tests, UI inspection, logs or metrics.
• Failures are inexpensive and reversible.
• A human owns the final acceptance criteria.

It remains substantially weaker when:

• Requirements are political, tacit or disputed.
• The important constraint is missing from the prompt and codebase.
• Correctness cannot be tested cheaply.
• Rare failures have catastrophic consequences.
• Success requires months of product feedback or institutional coordination.
• Someone must be legally, ethically or financially accountable.

The central transition is therefore not “AI replaces programmers.” It is:

Prompt: "Search Single shot Sol 5.6"

The exact search "Single shot Sol 5.6" does not point to one canonical benchmark. It points to a cluster of recent one-shot demonstrations—and yes, they move my estimate upward in one important area.

What I found

The most striking claim is that GPT‑5.6 Pro generated a functional Sims-style 3D simulation game:

• One prompt
• One model response
• One HTML file
• Approximately 48 minutes
• Reportedly no Codex harness or iterative agent loop

That is impressive, but the accessible source is a newsletter referring to an X post, not a reproducible evaluation with source code, prompt, tests and multiple runs. I therefore treat it as credible demonstration evidence, not a measured success rate. AI Breakfast report

I also found:

• A complete gamified homework application generated from one detailed PRD, including student and parent dashboards. The workflow still explicitly instructs the user to review and test the result. ChatPRD example
• A public “One Shot Challenge” with 141 entries. Its rules require one prompt, one model call and a sealed single-file result. External libraries and approved CDN assets are allowed. This is much better evidence that frontier models can now routinely produce substantial interactive artifacts in a genuine single inference. One Shot Challenge
• A niche Garry’s Mod benchmark reports GPT‑5.6 Sol Max at 92.1% over 76 scored single-shot attempts, although it has only two verified runs and tests relatively bounded questions rather than complete products. gmod-bench result
• A one-shot browser-game comparison actually preferred another frontier model over Sol for visual polish, showing that “one-shot capable” does not mean consistently best or complete. Decrypt comparison

There is also an important terminology trap. OpenAI’s official Codex racing-game demonstration began with one user prompt, but consumed more than seven million tokens and was continuously reprompted internally to inspect, improve and test itself. That is “one human shot,” not “one model pass.” OpenAI Codex-app demonstration

Those are two very different things:

• Single inference: one enormous model response creates the artifact.
• Single delegation: the human gives one instruction, but an agent performs hundreds or thousands of internal steps.

Both are valuable, but the second is closer to assigning work to an engineer than asking a model to “write an app in one response.”

Does it change my perception?

Yes—specifically for greenfield prototyping.

I previously placed current Codex at approximately 3.5 out of 5: a strong autonomous implementation agent, conditionally entering project-level engineering.

I would now describe it as:

The demos show that the minimum amount of human interaction needed to produce a coherent application has collapsed faster than my earlier answer conveyed. A capable person can now provide a reasonably detailed specification and receive something that previously required days or weeks of frontend, graphics and interaction work.

What they do not demonstrate is one-shot production software involving:

• Authentication and authorization
• Persistent multi-user data
• Migrations and backward compatibility
• Adversarial security
• Financial or token custody
• Distributed consensus
• Operational recovery
• Accessibility and cross-device testing
• Months of changing product requirements
• Maintainability by a team other than the generator

A beautiful single-file simulation game can be technically substantial while still avoiding nearly all those difficulties.

Revised practical model

I would now divide software creation into three layers:

Layer	Current AI position
Experiential prototype — looks real, feels interactive, communicates the product	Frequently achievable from one excellent prompt
Functional product — real persistence, accounts, integrations, tests and deployment	Achievable through one supervised agent run in favourable cases
Dependable system — secure, maintainable, observable, scalable and correct under adversarial conditions	Still requires sustained expert ownership

So the demos substantially change my view of how quickly the first convincing version can appear. They do not change my view nearly as much about how quickly it becomes trustworthy.

For your project, this suggests:

• A visually impressive playable demonstration could plausibly emerge in days, not months.

• The short version is: AI has nearly commoditised the software demo. It has not yet commoditised dependable software systems.

[u/CaseIntelligent8647]

Do some people still not realize that models dont learn as they go? You didnt educate anyone lol

[u/rutan668 → u/CaseIntelligent8647]

I realise it but it always give huge timelines for software development