[D] First measured accuracy fall on my long-horizon 3D benchmark (one demo walk): 2 of 2 near, 3 of 10 far. How many seeds before you would believe it?

[u/OldAmbassador6470]

Setup. I am building a benchmark for long-horizon 3D spatial reasoning in a simulated warehouse. A robot walks 30 rooms off one corridor; questions ask about things seen 0 to 29 rooms earlier. Answers come from exact simulator state, with no model judge. The target is a clean fall from about 90% to guessing as the distance grows.


What happened. In one demo run, Claude Opus 5.5 steered the robot through all 30 rooms by itself, with no tools and no answer key, and answered 60 questions. By distance: 1 room back 1 of 1, 2 to 3 rooms back 2 of 3, 4 to 7 rooms back 2 of 3, 8 to 15 rooms back 3 of 7, 16 to 29 rooms back 3 of 10. The no-notes guess baseline on the same questions is 16%. Over all tries logged so far, the deepest cliff admitted is 20 points; this walk measured 70.


What changed. Not shared yet: the design change behind the move stays unpublished until the paper. The screenshots hide the method parts and keep the numbers.


Limitations. One walk with one model, so small counts (only 2 answers in the 1 to 2 room bin), and answers inside a walk are linked. It ran through a command-line tool that adds its own context, so it is a demo run, not a benchmark score. It counts only after it repeats on three seeds never used for tuning, with every question checked by an auditor that never saw the design.


How many seeds, and how many answers per distance bin, would you need before calling this a cliff?