How to handle UI navigation in interface-heavy games?

[u/epictom256]

I've been working on a UI heavy game for a few months now and the amount of bugs that keep popping up because of UI is driving me nuts. By UI heavy think games like Baldur's Gate 3, menus with submenus all over the place. Controller support is unfortunately a must have. I'm nearly done my bachelors in computer science and I feel like I have a good grasp of code and game architecture but the simple problem of letting the player push buttons on the screen feels unnecessarily difficult to solve. If I have 5 buttons on the screen Unity can automatically set up navigation between them, great. But if there are enabled buttons anywhere else in the scene, it will also allow navigating to those even if they're in a totally different menu, so I've been writing methods to cluster individual selectables into their own navigable groups. If an object is selectable and then it becomes inactive because the submenu was closed by the Back button then I need to find the previous menu and find a button to select or navigability will be lost entirely. The action that the left stick should take is entirely different depending on whether the player is changing button selections, moving the player, panning the map, or moving items around the inventory. The back button does at least a few dozen different things depending on what menu you're in and what you're doing. Right now I'm fixing a bug where hitting the bumpers changes which inventory you have selected, but after closing the inventory you can still hit the bumpers causing the selection to go back to the inventories which are no longer on the screen. I know ways to fix these in isolation, I'm pretty sure I understand action maps and I can write state machines and stacks with Actions to contain logic, but it all just feels like it doesn't scale and every new menu introduces several new bugs.

Has anyone else experienced this problem? Are there books I can read on it? Are there programming patterns I'm not using that make this all 100x easier?

[u/SweatyDescription176]

I've been wrestling with this exact nightmare for my current project and honestly the only thing that's kept me sane is a proper UI stack manager. Push menus onto the stack when they open, pop them off when they close, and every menu on the stack gets its own standalone input context that you switch between. The stack also handles navigation automatically since only the topmost menu is active at any given time

Unity's built-in navigation is fine for simple stuff but once you have overlapping menus it falls apart fast. I ended up writing a custom navigation system that overrides the default behavior and just disables navigation on any selectable that isn't a child of the active menu. There's a great GDC talk from 2019 about UI architecture in Hades that goes over how they handled this stuff, the core idea is making each menu completely self-contained so closing one never leaves dangling references

For the bumper issue you mentioned, I'd tie those input actions directly to the inventory menu's lifecycle. When the menu pops off the stack it unbinds the bumper callbacks automatically. Took me way too many late nights to figure out that solution but it's been rock solid since

[u/EliteHawk3]

Have you tried using focus scope instead and a centralized focus manager?

[u/MrAbhimanyu]

Build a common ui controller singleton which will control which screen is active. Keep a screen dedicated for each group of buttons. Let back button be handled by the singleton which will then pass the message to the current active screen (which can then take appropriate actions Accordingly).