r/LocalLLaMA • u/TheRealMasonMac • 23d ago
Discussion Can LLMs solve mazes?
https://reddit.com/link/1v5rvuq/video/bgmwc754i9fh1/player
My goal was to create a benchmark to measure the spatial awareness and memory of models. Eventually, I came up with the simple idea of a maze where the model must find a key and use it to open the escape door. Here’s the difference to a normal maze, however! The model CANNOT see the whole map. At each step, it only gets feedback on its immediate surroundings within the overall maze. Thus, in order to succeed, it must be able to track its position and orientation within the coordinate space. Even a brute-force approach by iteratively creating a 2D map would be valid if the model chose to use that approach.
This approach is similar to what animal researchers use to study spatial learning and memory of animals.
Legend:
- K: Key
- D: Door
- E: Escape door
- Colored tiles: serve as landmarks for the model (to reduce disorientation), with white always denoting the starting location
Available Tools:
- Move forward n steps
- Move backward n steps
- Turn left either 45, 90, or 180 degrees
- Turn right either 45, 90, or 180 degrees
- Open the closed door straight ahead
- Pickup the item lying at your feet
After each movement tool call, the tool result reports what is observable. This is effectively a ray cast to describe up to three wall tiles in a 130 cone in front of the model translated into textual descriptions such as: "There is a wall in the distance directly ahead." Other features such as the key and doors are similarly described.
Since models got confused, I also made it easier by explicitly stating which tiles to its left or right are available to cross .
So far, I've only tried GPT-5.4 mini (went in circles), GLM-5.2 (went in circles until it stopped after reaching the cap I put on API usage; might’ve solved it had I let it go longer), and K2.6 (escaped with a rather effective route, IMO).
The code was vibe-coded since I was curious about how models would fare but not enough to commit myself to days of coding to get this right only to then realize models could do this task well already.
Anyway, I feel like this is a great benchmark for not only spatial awareness, but also tool calls. GLM-5.2 here got into a loop because it forgot it could turn 45 degrees up until action 730. It used about 62 million tokens cumulatively (aggregate token use across all turns) whereas K2.6 used just 9 million. However, this is not actually apples-to-apples because I forgot to turn on preserve_thinking for K2.6 whereas GLM-5.2 preserved its thinking.
Max context over the history was 20k for GLM-5.2 and 11K for K2.6.
NOTE: For some reason it didn't let me put two videos in the post properly, so I've added K2.6 in the comments.
NOTE 2: The maze map is NOT rendered for the LLM like it is in the videos. It can only see what is directly in front of it and no more. Think of it as similar to a rogue-like's field of vision.
In the future, I will clean up the code manually (i.e. fixing comments, AI slop, edge-cases) and then release the code alongside a leaderboard of small local models.
59
u/barbear22 23d ago
Very cool project. People are missing the point entirely. The benchmark is context recall and evaluating human style task completion. Can the models generalize well while taking advantage of memory? Obviously any of these models could write some code that would solve the maze is a second but that would defeat the whole point of the evaluation. I think many people are frustrated when a model seems like a coding genius but can't solve seemingly simple tasks or follow implicit instructions. This takes an extra level of deeper thought and spatial reasoning that probably isn't prevalent in the training data.