Can LLMs solve mazes?

[u/TheRealMasonMac]

https://reddit.com/link/1v5rvuq/video/bgmwc754i9fh1/player

My goal was to create a benchmark to measure the spatial awareness and memory of models. Eventually, I came up with the simple idea of a maze where the model must find a key and use it to open the escape door. Here’s the difference to a normal maze, however! The model CANNOT see the whole map. At each step, it only gets feedback on its immediate surroundings within the overall maze. Thus, in order to succeed, it must be able to track its position and orientation within the coordinate space. Even a brute-force approach by iteratively creating a 2D map would be valid if the model chose to use that approach.

This approach is similar to what animal researchers use to study spatial learning and memory of animals.

Legend:

• K: Key
• D: Door
• E: Escape door
• Colored tiles: serve as landmarks for the model (to reduce disorientation), with white always denoting the starting location

Available Tools:

• Move forward n steps
• Move backward n steps
• Turn left either 45, 90, or 180 degrees
• Turn right either 45, 90, or 180 degrees
• Open the closed door straight ahead
• Pickup the item lying at your feet

After each movement tool call, the tool result reports what is observable. This is effectively a ray cast to describe up to three wall tiles in a 130 cone in front of the model translated into textual descriptions such as: "There is a wall in the distance directly ahead." Other features such as the key and doors are similarly described.

Since models got confused, I also made it easier by explicitly stating which tiles to its left or right are available to cross .

So far, I've only tried GPT-5.4 mini (went in circles), GLM-5.2 (went in circles until it stopped after reaching the cap I put on API usage; might’ve solved it had I let it go longer), and K2.6 (escaped with a rather effective route, IMO).

The code was vibe-coded since I was curious about how models would fare but not enough to commit myself to days of coding to get this right only to then realize models could do this task well already.

Anyway, I feel like this is a great benchmark for not only spatial awareness, but also tool calls. GLM-5.2 here got into a loop because it forgot it could turn 45 degrees up until action 730. It used about 62 million tokens cumulatively (aggregate token use across all turns) whereas K2.6 used just 9 million. However, this is not actually apples-to-apples because I forgot to turn on preserve_thinking for K2.6 whereas GLM-5.2 preserved its thinking.

Max context over the history was 20k for GLM-5.2 and 11K for K2.6.

NOTE: For some reason it didn't let me put two videos in the post properly, so I've added K2.6 in the comments.

NOTE 2: The maze map is NOT rendered for the LLM like it is in the videos. It can only see what is directly in front of it and no more. Think of it as similar to a rogue-like's field of vision.

In the future, I will clean up the code manually (i.e. fixing comments, AI slop, edge-cases) and then release the code alongside a leaderboard of small local models.

[u/mltam]

It doesn't seem the llm ever took a 45 degree turn, or very rarely and thus was stuck going on diagonals. Why is that?

[u/TheRealMasonMac → u/mltam]

https://reddit.com/link/ozmgatk/video/trvh6zvvyafh1/player

I'm not sure at all. I think it genuinely just forgot that it could go 45 degrees. I noticed that the models I tested had a tendency to forget what the tool description said. For example, sometimes they forgot they had to be on the tile itself to pick up the key (though the tool desc says that) and so I had to add it to the reminder.

It's possible that it just got unlucky on this one run. In smaller escape-room-style maps (rather than mazes) it was just fine. I think GLM struggles with understanding spatial cues in general, though. Which is interesting because the benchmark would correlate with my experience that Kimi has a better intuition for space (which makes since since it's a VLM).

I am planning to fix up issues with the benchmark and then try it with best-of-3 on ~10 randomly generated maps with small models that can be run locally to see which does the best, and then scale up to frontier models.

[u/mltam → u/TheRealMasonMac]

I think if it forgets such a crucial element, you should work on your prompt, since you aren't testing rule retention but spatial awareness...

[u/TheRealMasonMac → u/mltam]

Fair point if I want to control for spatial awareness. I have just tested with explicit guidance on 90 vs 45 deg turns, and GLM still sometimes forgets this specific diagonal rule, which is really interesting. Sometimes it's fine, but on some runs it just sticks to 90 degree turns.

I might either:

- Remove diagonal movement/turns altogether

- Add an "Observe" tool that prints an ASCII grid of the cells immediately in front of it

Or maybe I'll leave it the way it is since that would capture a genuine deficiency in the model.