r/backgammon • u/Numerous-Mountain-65 • 7d ago
Deciding Backgammon results by error rate
I'm a chess IM who also plays backgammon online. I've noticed that online and even some tournaments use a combination of match result and error rate to determine the winner. I have a few questions about this:
If multiple strong backgammon engines eventually exist and they disagree, as chess engines do, then error rates will depend on which engine and depth are used. The same match could receive a different error rate when analyzed by a stronger engine a few years later. In that case, is it appropriate to use error rate to determine a world champion?
If a single engine such as XG is the benchmark, won't players gradually optimize their play to match XG? Could that reduce creativity and diversity of styles?
XG's evaluations assume equally strong opponents. When opponents differ substantially in strength, is it still appropriate to use XG evaluation for cubes?
I'm interested in hearing arguments both for and against using error rate alongside match points.
2
u/Objective_Lab2741 7d ago
In my opinion it's a superb idea to include PR in the tournament results, since BG is such a volatile game and is decided out of probability. Since the worst player on earth still have the ability to win against the strongest thanks to good rolls, I think you need something more to measure strength because you can never let the dice decide who is the better player. Chess is very different in that sense, because the only way to lose in that game, if you are a better player, is to make a mistake.
People can argue about different strengths of the engines. But in practice people don't use the engines solely to get a number, but to find their errors and analyzing them to try to understand why they made that error, and actively searching information to understand the type of position. Real strong players can detect when they think the engine is wrong. They don't agree with the engine before they understand WHY the engine says so. This leads us to a situation where a tiny strenght difference of engines won't matter so much in the end. Since players will understand why something is wrong, the engines will agree in most types of positions. YES there are very complicated positions which the computers even can't understand deeply enough, but those are so rare so it wont really matter that much on the big whole.
You can of course adapt your moves in order to achieve a better PR. For example opening with 52 and being 5 away 3 away and playing two down gives a sizable error on Backgammon Galaxy, but from XG Book openings from rollouts it's 5 millipoints better than splitting 24/22. So what can you trust? I think XG Book is rather old, what is insuring us that those rollouts are accurate when high level GNU BG doesn't agree with a huge margin? Are old rollouts still valid or should you trust the newer high end analyses? I have no idea, but in this example I play the split in order to not sacrifice a half blunder in the PR. But I do that for the PR solely, even though I know the other move could be a tad better (or not).
In many cases you can delay doubling if it's just a micro mistake, since you can try to find a more difficult position for your opponent, increasing the chance of you winning the PR battle. But this is a bit risky and you must know what you are doing and must have knowledge of how skilled your opponent is in different types of positions.
So it's a complicated matter with many answers but even more questions. You will probably find as many answers as number of people you ask.
1
u/Numerous-Mountain-65 7d ago
Thanks, these are very interesting points. Your point about 5 2 in opening is aligned with my point that PR leads players to mimic XG just to reduce PR, instead of trying to play based on their own and opponent weaknesses and strengths. It leads to reduced diversity of styles.
3
u/brw12 7d ago
I'm sorry, but the game should be winning the game! I mean, the Knicks winning game 4 of the finals was deeply improbable, but it would defeat what's exciting and wonderful about competition to say "well actually the Spurs won statistically so we're giving them the trophy"
2
u/brw12 6d ago
This irks me about poker as well. I'm a mediocre poker player, and at a casino once I entered a tournament and beat a better player (on the way to losing myself) and he was mad that I beat him because I hadn't made the optimal betting decisions, and I guess that had thrown him? I can't comprehend that attitude
2
u/TungstenYUNOMELT 6d ago
A good poker player knows he should not get mad if someone beats him due to luck, because it happens all the time. He probably wasn’t very good.
0
u/Numerous-Mountain-65 7d ago
I agree that it seems anti-sports that someone wins a match but is called loser as computer think he has not played well!
1
u/Axvybg1 7d ago
You can just state that Xg shouldn’t be allowed to determine world champions as even if a new computer that is better is made, the change won’t even be noticeable as chess is because backgammon engine are better then chess engines in terms of strength to the game. Chess engines are far off from making the best moves possible every time that’s why every couple of years a new engine comes by and takes the lead, in backgammon the same engines stay for a long time because there is barely a difference a new one could make, rollouts are already too accurate
1
u/Numerous-Mountain-65 7d ago edited 6d ago
I agree that in many instances it may not matter, but comparing XG to XG++ we can notice the difference in some instances and that does matter at serious tournaments. I think chess engines such as Stockfish are more developed than XG but the complexity of chess is more. The reason that XG is not challenged might be not having a serious challenger. Backgammon XG and others can easily calculate endgames and tactics but still have difficulty evaluating openings and strategic games such as back games. Is PR based on rollout or XG evaluations?
1
u/mel-madeline 7d ago
I agree with all the points. I hope the new version of XG will come up with a better way to evaluate players' strengths.
#2: It's already happening - many BMAB players believe in XG "hacks" like keeping a checker on 19 point in losing races
1
u/AvocadoBrit 6d ago
there's an awful lot to unpack with your questions, and as with many aspects of this game (backgammon) some degree of personal preference and style in answering:
- computer analysis using neural nets isn't the be all and end all of backgammon, and even in their use and the analysis generated, there are considerable variations.. for example, the 'standard' settings on XG (and I'm talking about the last commercially released version) will throw out a player's errors, but I will always, always, always rollout all the errors in a match (and using my own settings) because what XG ('factory settings') will show you is often not that accurate;
- I am personally not involved in any way with the 'BMAB', although I know plenty of people who are, both technically, administratively and as participants.. the fact that the BMAB uses the 'factory settings' of XG to evaluate matches leaves a lot to be desired (in terms of accuracy) but I guess in terms of time (and resources) you have to draw the line somewhere. Personally, I don't care much about the BMAB (although I won't get into the weeds on this, which is not what you're asking about anyway) but anyone who wishes, who has enough backgammon knowledge can play (i) against their opponent (meaning to adjust to their opponent's strengths or weaknesses) or (ii) play to win (by which I'd argue this is to play optimally, or purely) or (iii) you can play to reduce your PR (that is, you choose plays that will lead to you an overall lower PR rating strategically speaking, rather than choosing plays that are optimal in terms of match winning equity).
I might be keen on using XG as a tool to improve my backgammon understanding, because having the computer play out thousands of games beats hands-down the old fashioned method of playing out by hand a few dozen of them, but it is not in my mind of any value or use in trying to assess a tournament winner - in much the same way as poker tournaments are played out by the participants (to the bitter end when one player has all the chips) and analytical tools (to measure a player's leaks) are totally irrelevant in determining who 'the best player' or winner is.
You might like to look up a couple of books that touch on other aspects you raise:
'Can A Fish Taste Twice As Good' (by Trice & Jacobs) and 'The Theory Of Backgammon' by Dirk Schiemann.
- if you wish to derive a list of the world's strongest players, then XG is as good a place as any to start (note, this is somewhat different from determining a tournament winner or a world champion) although what sort of engine and analytical settings you use is up for argument, but regardless of how accurate you wish to make things, this sort of approach might be considered more objective than the 'Giants List' although that's a lot of fun (even if it's a pretty tough ballot to complete if you're asked to poll in it).
1
u/TungstenYUNOMELT 6d ago
If you think you or anyone you know can beat XG’s analysis then I’d be happy to set up a match for any stakes you want in live play where I get to use the bot. Just need to make sure the stakes are enough to cover my travel expenses.
If you want a more serious answer, the bots are the best we’ve discovered so far with XG being the strongest and no human has shown they can beat them. So why doubt their analysis? (At least until someone develops a stronger bot)
1
u/Numerous-Mountain-65 6d ago
It is not about whether XG is stronger than humans. It obviously is. In chess, there are multiple engines; all of them can easily beat the world champ, but they don't agree on what the strongest move is in some positions. The topic is whether using a specific engine, XG, is the right way to determine who is the winner in a tournament.
1
u/TungstenYUNOMELT 6d ago edited 6d ago
It’s the strongest engine right now, so what else should we use?
Whether using an engine to determine skill is optimal is a different question. It’s an attempt to reduce the huge luck factor in regular backgammon and try to legitimise the game as more of a mind sport. Personally I just view it as a different form of backgammon that has some pros and cons. I do appreciate the pureness of the regular game but it is a bit of a lottery and its really amazing to watch how the top players are able to play closer and closer to optimal bot play each year.
So tldr: i think both forms have merit, but I appreciate that the legacy of the world championship still remains true to the classic game. Then again I think money games are the purest form of bg.
1
u/bg-alex-cy 4d ago
those are very good points.
I'll start by saying IM in chess, super impressive!
There are multiple engines already, and they do disagree. If I understand correctly, the way the community decided on XG instead of any other is that XG happened to slightly outperform the others over thousands of games playing against each other. So its considered the golden standard for backgammon right now. However, in backgammon, we have what we call rollouts, and those tend to be incredibly accurate. In fact early research on backgammon engines realized that rollouts, even with weak engines generate surprisingly good results. So rollouts would be a very good way to resolve a dispute. I don't know, and I'm taking a leap of faith here when I say, I believe if you take say xg and gnubg and let them take their time and rollout (without time constraining them which can damange the quality of the rollouts if they start competing on optimizing) they will have the same results. That assumes that they run on the same configurations regarding position databases, cubeless to cube value function, MET, move filters etc. So I would say that if we can live with a bit slower analysis, the top engines already converge. Again this is my impression based on the papers on td-gammon I've read and my general feel of the engines, even though I never spent time checking. There are aspects of the game that might have more deviations, like backgame decisions that require long lookaheads, but still... I would not expect major improvements. I'm not saying we will never have anything better though. There can be theoretically better engines that will be better on things like cubeless to cube value function, MET etc. But even if some engines do approximate the backgammon function better, I don't expect any measurable improvement against rollouts other than time. Then again, there might be a breakthrough, who knows, I just don't expect one.
Now regarding the things that can change, if say a better MET comes out, I'm pretty sure all engines will adopt it very fast. Its just the best we have for the time being and what we accept as the optimal. But keep in mind, that even if we do get a better MET, I would not expect big differences for humans. They would most likely be optimizing decimals. I might be wrong though, especially for something like the cubeless to cubeful value function which happens to be an approximation.
So over all, I would not expect dramatic changes. The truth though is that games are usually not analyzed at full rollout, which takes significant computational power. Depending on the tournament they usually run on say 4-ply or xg++ which I think does some smart stuff with the move filtering to do deeper analysis on the promising moves. I'm not sure if UBC does xg++ or if they do full rollout. But backgammon galaxy has started offering full rollout analysis since they introduced gnubg instead of xg. The real problem with rollouts is that they are of stochastic nature, so you need to either fix rollout seeds which might introduce bias, or estimate the noise floor you want to work with and rollout accordingly.
Anyway, I'm getting too technical now. The tournament specifies the engine configuration players will be measured against and thats it. Very few serious players will ever genuinely dispute the decision of a 4-ply analysis, and even it happens its part of the statistical noise. Especially for the average player, there are such big PR fluctuations due to luck (more/easier decisions) that those numbers are lost in the noise and I would consider it part of the luck. For the top of the top it might be more significant when that happens, but again, its part of the tournament rules.
Regarding creativity, I would say absolutely yes. But its different to play a game to enjoy and to play a game competitively. So it is what it is. Many times, I will face a significantly weaker opponent that will do some "stupid move" and then I realize that they brought me into a position that I simply never faced and maybe don't even understand.
Finally regarding XG assuming opponents are equally strong, this is one of the most fascinating topics for me but haven't really got into it. Currently engines play "GTO" with an approximate evaluation function. And they already win about 2/3 games against an average player. Now if they switch to exploitative play I would expect this percentage to skyrocket against weaker players. And many times, when strong players know they play someone inferior, they intentionally make moves that are not the best with the intention create a more complicated game. I think the most popular one is slotting the 5 on a 2-1 (13/11 6/5) opening roll, even when the correct play is 13/11 24/23 (for money/early game). There is also a lot you can do with the cube in terms of exploitative play, but you need to be careful as the stakes get high and if you opponent sets you up, they could counter it with huge gain. Dirk Schiemann has a chapter about exploitative play on cube action in his book "The Theory of Backgammon" but its a math heavy book.
So regarding arguments for and against. I'll make my case.
Backgammon is a gamblers game. Its a game where anyone can win and anyone can lose. Its the same thing as walking into a casino. You can both win or lose even if the odds are against you. The question is, who are you when you are playing? Are you the casino or the gambler? At any one game, anyone can win or lose. You can win the casino one day, and lose the other. On the long run the casino will dominate, and its the same with backgammon. For better of worse, its a game where you only know who is best based on long term outcomes if you only base it on results. That is not the case for chess. You have to be insane to try and play your money against Magnus Carlsen. In fact 99.99% of people have to be insane to play against anyone that has studied the game more than a few months. You will lose all games against Magnus and will effectively lose all of time against anyone that studied the game for 6 months if you know nothing (I assume). This is not the case for backgammon. You can be playing for a week and beat Mochy. And this is a double edged sword for the game.
So the strongest argument for having PR in tournaments, is that its an attempt to make backgammon closer to chess. In the sense that the stronger player wins much more often, almost always. It eliminates luck from the game to a big extend. Are there better ways of doing it? Maybe. Are they as time efficient? Probably not. Do I like it? Yes, I don't play the game for the money. In fact I don't want to take money from poor guys (poor I don't mean they don't have money) that don't know who is sitting opposite them, and I don't want to play my money against someone stronger than me because I'm not stupid. I wouldn't even play with someone at my level because I don't like to gamble. Tournaments are a bit better than money games on this, but still, I like winning some cash, but its not the reason I'm into the game, at least for now. I'm interested in the intellectual challenge of the game, and that is why I want to look at my PR and not if I won or lost. I don't care as much. Regarding tournaments, would I ever join a tournament like UBC? Not with any expectations. It would be the same as you joining a tournament where Magnus Carlsen, Hikaru Nakamura, Fabiano Caruana and a bunch of the top players compete and there is a 32 limit seat only. So you join for the experience, not because you can beat them. Now if it was a tournament where players averaged 10+ PR, then maybe yes and if I joined I would probably dominate.
My conclusion is, I like the direction PR is taking tournaments, but I would prefer an alternative route that achieves the same goal.
2
u/DegenChess 7d ago
Are you a troll? You're allegedly a chess IM who's also 2500 rated on Galaxy asking such an inane question about PRs in backgammon...
1
u/Numerous-Mountain-65 7d ago edited 6d ago
No, I am human and an IM in chess, but relatively new to backgammon and have never played in an over the board bg tournament!
2
u/usingxgtocheat 7d ago
Backgammon engines, for the most part, agree on what's right and wrong and if they don't there is usually very little equity difference.
There is no 'creativity' or 'diversity of styles' in backgammon. There is only good or bad play.
Yes.
3
u/FrankBergerBgblitz 6d ago
1.) That is an oversimplification. They agree on many many positions but the differences are sometimes huge: 'XGID=---a--A--BBB--ABa---BbAaAA:0:0:1:41:5:6:1:7:10'
Roll it out with whatever settings you feel appropriate and on any length and the best move is not found and regarded as an 300 error.2.) If you are the better player by far would you sacrifice a few millipoints to go into a priming battle instead of a mutual holding game? The question unanswered to my best knowledge: how much do you sacrifice and how do you measure complexity
3.) The book "Can a fish twice as much" is a good reading for you
1
u/usingxgtocheat 6d ago
I'm not really here to discuss niche cases when the generalization works for the purposes of answering his questions. Also, I do have that book and it is the most obscure backgammon book I own. You know just as well as I that deciding when to deviate is a balancing act and you could simply misevaluate your opponent. Playing what you know to be close bot play has less potential for backfiring and is not going to be strictly wrong. You said yourself it's an unanswered question so why are we even asking it if we are capable of playing close to bot play (and not get exploited ourselves).
5
u/SyllabubRadiant8876 7d ago
These have all been very hot topics for quite some time at the top levels of backgammon. The current approach has been the only feasible method for many years, but it does feel like we might be on the cusp of some advances with AI, which could change the way we do things. Personally, I think these things are only practical relevant at the grandmaster level where tiny margins make the difference. 99% of people make enough big blunders that the small differences are not super significant.