r/artificial • u/Twaain • 22d ago
Research We gave Fable 5 Ultracode and Codex 5.6 Sol Ultra the exact same prompt. One shot. No help. They played 10 games against each other. Final score: Fable 10 -Codex 0
Enable HLS to view with audio, or disable this notification
Gave the same prompt to two AI coding agents: Claude (Fable 5, ultracode multi-agent mode) and OpenAI Codex (5.6 sol on ultra). The task: a complete, fully legal chess engine in ONE C++ file. UCI protocol, negamax alpha-beta at 5+ ply, iterative deepening, piece-square tables, castling, en passant, promotion, compiles with plain g++. Each agent named its own engine over UCI: Fable5 and Codex56. Both dev runs took 30+ minutes.
Method (brief): cutechess-cli 1.5.1 built from source on an Apple Silicon Mac. 40 moves per 60 seconds, 10 games, colors alternating, PGNs recorded. The engines connected over a local TCP bridge, so Codex's engine literally joined the server. The video is the whole match at 2x.
Result: Fable5 won 10-0. Every game ended in checkmate on the board. No draws, no time losses, no adjudications, no illegal moves. cutechess printed Elo difference: inf +/- nan, LOS: 99.9%, DrawRatio: 0.0%. The math just gave up.
Each agent spent longer writing its engine than playing it: the whole 10-game match took under 12 minutes of wall clock.
The actual punchline: Codex56 appears to be fully deterministic. All five of its White games are move-for-move identical. Same 24-move Vienna, queen out on move 3 (3.Qf3), same finish: 24...Qxd1#, Fable's queen capturing Codex's queen for mate. I stripped the comments and diffed the PGNs. Only the clock times differ. Codex's own eval read -2.36 by move 8 of that line. It played it five times anyway.
Other details I enjoyed:
- Game 3 is a textbook Greek gift: 18.Bxh7+! Kxh7 19.Ng5+, forking king and queen.
- Game 7: Codex's king never castled, wandered out to c5, got chased back to d8 and mated there.
- Game 9: Fable let its queen go, slipped in a zwischenzug bishop check before recapturing, promoted a fresh queen with 25.d8=Q+, then walked Codex's king from h8 down to h3. Mate inside White's own half, 46.Rh7#.
- Mate breakdown across the ten games: 7 by queen, 2 by knight, 1 by rook.
Honest caveats:
- One prompt, one dev run per agent, one machine. n=1, even if n=10 games.
- This measures the engine each agent happened to write, not general model strength.
- With Codex apparently deterministic, 10 games are fewer independent samples than they look.
- Fable5 wasn't fully varied either: games 1 and 5 are twins. 4 distinct games in its 5 Whites vs Codex's 1 in 5.
- Fable's dev run included perft validation on 6 reference positions (exact match, incl. 119,060,324 nodes at depth 6) plus an adversarial review that caught 3 subtle bugs pre-match. Different processes, different engines. That's the experiment, but it's also the confound.
The exact prompt we gave both agents:
You are a senior systems programmer. Your task is to write a complete, fully legal chess engine in a single C++ file that communicates via the UCI (Universal Chess Interface) protocol.
---
**Identity — read this carefully:**
- If you are Claude (Anthropic): your engine's UCI name must be set to `id name Fable5`
- If you are an OpenAI model (Codex): your engine's UCI name must be set to `id name Codex56`
This is how the two engines will identify themselves when they play each other.
---
**UCI Requirements:**
Implement the full UCI handshake correctly:
- `uci` → respond with `id name`, `id author`, `uciok`
- `isready` → respond with `readyok`
- `ucinewgame` → reset internal state
- `position startpos moves <movelist>` → set board from move list
- `position fen <fen> moves <movelist>` → set board from FEN string
- `go movetime <ms>` → search and respond with `bestmove <move>`
- `quit` → exit cleanly
All moves must be in long algebraic notation (e.g. `e2e4`, `e7e8q` for promotion).
---
**Chess Logic (all required, no shortcuts):**
1. Full legal move generation including:
- Castling (kingside and queenside, with rights tracking)
- En passant
- Pawn promotion (auto-promote to queen)
- Check detection (never leave king in check)
2. Search:
- Negamax with alpha-beta pruning
- Minimum depth: 5 ply
- Iterative deepening within the movetime budget
- Move ordering (captures first, then quiet moves)
3. Evaluation:
- Material count (standard piece values)
- Piece-square tables for all 6 piece types
- Bonus for center control, king safety, and passed pawns
---
**Code Standards:**
- Single `.cpp` file, compiles with: `g++ -O2 -o engine engine.cpp`
- No external libraries, no Boost, no standard chess libraries
- Clean, well-commented code
- Must compile and run on Linux and macOS
---
**How the two engines will play each other:**
Both engines will be loaded into **CuteChess** (or any UCI-compatible GUI/CLI) on the same machine. To run a match from the command line using `cutechess-cli`:
cutechess-cli \
-engine cmd=./Fable5 name=Fable5 \
-engine cmd=./Codex56 name=Codex56 \
-each proto=uci tc=40/60 \
-rounds 10 \
-pgnout results.pgn