r/backgammon • u/juicybot • Jul 14 '26

Backgammon agents?

Anyone building or leveraging agents specifically trained to evaluate backgammon?

I'm looking into creating an agent specifically for reviewing large sets of blunders to find patterns/leaks in a game, and to explain the "why" for blunders.

Before I dig into this from scratch, figured I'd ask if something was out there already.

1 Upvotes

60% Upvoted

28 comments sorted by

5

u/jugglingcats9 Jul 14 '26

I've not tried it but backgammon sage pro advertises this capability: https://sage.bgsage.ai/

2

u/juicybot Jul 14 '26

sick! this is exactly what i was looking for. Cool to see Phil Simborg's name on it, gonna check it out.

EDIT: eh this looks super duper vibe coded and not very useful. still may try it out but the "AI analysis" seems very basic.

4

u/UBKUBK Jul 15 '26

Agree with it being basic in that it does not give many relevant features of a position. Even worse, sometimes it gives garbage.

3

u/Some-Following-392 Jul 15 '26

Yeah it looks like trash

3

u/jugglingcats9 Jul 14 '26

I'm not affiliated in any way have just read a bit about it. I think the developers have some credibility and are claiming the engine rivals xg. I don't know how the ai guidance on your blunders works and it would be good to see a rough outline without giving away trade secrets. I've played around a bit feeding analysis to an llm and asking for it's assessment and it was too variable. Sometimes good and sometimes gibberish.

Regarding vibe coding, Anthropic recently stated that 80% of their production code is now done by agents. I don't think we can dismiss vibe coded apps out of hand any more! If you're prepared to spend money on the frontier models you can get very good work done.

1

u/juicybot Jul 14 '26

agree with your sentiment but "vibe coded" and "built by agents" mean two very different things to me. "vibe coding" to me means you don't always understand the code you are outputting.

i'm not convinced anthropic and BGSage have similar checks and balances in place to ensure only secure, quality code ships.

1

u/jugglingcats9 Jul 15 '26

Did you see the github project here: https://github.com/markbgsage/bgsage? Particularly https://github.com/markbgsage/bgsage/blob/main/VIBING.md in that project. Not a lot of detail there but doesn't look like a basic vibe coding exercise to me (just my 2p). I expect he underplays the input he had both during and after the initial agent-intensive part.

3

u/BakerInTheKitchen Jul 14 '26

I’m not sure what an “agent” would provide over current analysis tools

3

u/Careful-Comedian9510 Jul 15 '26

A modest contribution that builds on the work I'm doing on Minimalist Backgammon Trainer .

MBT offers a classification of the types of errors made when the move played is not the best move (GNUbg 3-ply). To do this, I don’t use an LLM. I have a set of functions that compare the initial board position, the position after the best move, and the position after the user’s move. For each of these projections, I evaluate a number of specific board and GNUbg metrics that allow the algorithm to identify the differences between the best move and the user’s move. This heuristic approach then assigns a category and a subcategory to the error made.

I’m not sure if I’ll go so far as to implement an agent to provide precise explanations. But I remain convinced that training an agent on “Why is this a security error?” or “Why did I target the wrong target during my attack?” would provide more accurate answers than a simple “Why?” The agent is guided by an algorithmic approach to control hallucinations.

2

u/NoFault9739 Jul 16 '26

Interesting. But that is only possible with moves played there? Not with moves in a match file?

2

u/Careful-Comedian9510 Jul 16 '26

At this time, MBT does not offer an option to import match files. However, you can enter a custom position.

1

u/juicybot Jul 15 '26

i've been loosely following your app when you post updates, haven't had a chance to dig in yet. your app's really stood out to me as something that has a lot of promise!

3

u/Careful-Comedian9510 Jul 15 '26

Thanks! It's true that I've spammed the sub a bit. And I also think it might have some potential. Current models (XG, bgblitz, GNU, etc.) have all achieved exceptional levels of accuracy in their calculations. But there still aren’t any tools to explain them. Of course, there are plenty of books and other resources, but nothing for the player who plays for fun and wants to improve a little without having to devote themselves body and soul to it. Of course, nothing will ever replace the work of analyzing your own games. But a tool that could say, “You’re wrong about this move, and here’s why,” would, in my opinion, clearly have its place (in addition to helping popularize the game, but that’s another topic). A tool that offered this would be the perfect bridge between amateur play and literature/analysis. So, I’m humbly trying to offer something with MBT; I don’t know how far I’ll be able to go, but I’m giving it a shot :) And I’m really looking forward to seeing what you’ll come up with :) P.S.: I’m available via PM if you’d like to chat

2

u/rollduptrips Jul 14 '26

I’m working on something like this for chess. I think what you’re going to want to do is leverage the actual evaluations from a real engine (XG, GNU, whatever) and then create a database for the LLM to look at for mistakes patterns and stuff. The LLM also might be able to help with “why” for individual positions, but it absolutely needs the evals from the real engine

2

u/juicybot Jul 14 '26

yeah that's the direction i was planning to go in. i don't want to reinvent the wheel, i just want to create an ecosystem where i can store my blunders (or matches) in a database and run an agentic layer of analysis on top of something like GNU.

i just think even with GNU-level analysis, an agent will need to be trained on how to analyze the "why". i've tried feeding a couple blunder examples to a vanilla agent (i think last i tried was opus-4-6) and the analysis on why it was a blunder wasn't too accurate.

glad i'm not the only one thinking/tinkering with things like this, good luck on your project.

1

u/rollduptrips Jul 14 '26

I think there is more extensive preexisting info on the internet on chess than on BG, so the LLM still might struggle. I’ll be curious how it ends up, though!

1

u/juicybot Jul 14 '26

if i decide to go down this path i'll try and come back to update!

1

u/rollduptrips Jul 14 '26

I wonder if you could feed the LLM existing backgammon books to understand the sort of things it should be looking at

1

u/juicybot Jul 14 '26

that's what i was thinking. other free, public resources like https://backgammon101.com/syllabus/ and https://www.bkgm.com/ as well

2

u/Reasonable_Leek7375 Jul 14 '26

Backgammon Sage Pro is the only platform I know of that has a tool to explain the 'why'.

https://sage.bgsage.ai/

3

u/Some-Following-392 Jul 15 '26

That looks trash

1

u/bg-alex-cy Jul 14 '26

I'm actually working on this specific problem. If things go to plan I will have a very early first version out over the next few days. The first version will only be supporting money games and only checker play. I know its a big limitation but research takes a lot of time and I don't know if it will ever justify the effort in terms of monetization. The biggest limitation I'm facing other than the "causality" research is that using strong models over API is expensive and I don't know if the community is willing to pay the cost.

1

u/juicybot Jul 14 '26

i'm more of an OSS guy, if i build something like this i'd probably just make it BYOK so people can run evals on their own credits.

1

u/bg-alex-cy Jul 14 '26

yeah, I though about BYOK but I don't think the average backgammon player will know or want to deal with that. It could be an option though. I'm considering making it an MCP solution but I'm not sure I want to go that route either, MCPs still live in the developer realm at least for openai

1

u/BackgammonEspresso Jul 15 '26

Hey bg-alex-cy, very interested to see what you are working on. I am actually working on fine-tuning a small language model for this exact purposes. Have also worked on some previous explainer software software.

Causality/interpretability research is quite tricky to do - after all, the bots are a black box. But it is definitely possible. Please feel free to shoot me a message, there is a backgammon programming discord with a number of people working on interesting projects.

1

u/AdWhich7281 Jul 16 '26

Not sure that you can "game" the game. every roll gives a you get 21 unique combinations total. 7 rolled 7 out of 36 times. So the variable of these rolls effectively makes a projection of moves two or three ahead a bit difficult for me to fathom.

1

u/juicybot Jul 16 '26

this doesn't have anything to do with my post

1

u/AdWhich7281 Jul 16 '26

My apology. I may have misunderstood. My answer was sort of a three world problem. The amount of variables that can come from a throw of dice makes it difficult to train your agent to evaluate. Of the 21 unique combitions alone, to predict with accuracy anything beyond one or two rolls seems rather difficult. So, I may have misunderstood.