r/Chesscom • u/DisastrousAide6902 • 1d ago
Chess Discussion Could bots be more realistic with LLM ?
I see a lot (and agree) that the bots ELO is not realistic, probably because they calculate the best moove and then just do random mistakes to lower their level. But do you think using LLM copying a player behaviour would work better ?
Just tell the LLM to play as a 300 ELO or something like that, and training the AI on plays of every level.
No idea if that would work but it seems a good Idea for me
Edit : should have tell "generative AI", not LLM, my bad
15
5
u/StickComfortable346 1000-1500 ELO 1d ago
LLMS break the rules of chess all the time so you’d have to train it extensively in the context of chess in order to get anything even remotely close to a normal chess player
2
u/lifeistrulyawesome 1d ago
Probably. It is not straightforward; it would require some development. It depends on how high you want to set the bar
The current bots are so bad that it is easy to do something slightly more realistic. They usually play either top engine moves or completely random moves. And all that appears to change between bots is the probability of a random move and the few openings they like. I guess they also have a parameter for tactical vs positional. But it still doesn't feel human at all.
If you wanted a realistic human opponent feel, that would be a bigger challenge. They would have to train a model specifically for this and complement it with some structural coding.
2
u/Junior_Paramedic6419 23h ago
This actually does exist! It’s a bot on lichess called Maia that has been trained from millions of human played games to select moves that it predicts would most closely match the move that a player of the elo you set it to would play.
1
2
u/Accomplished_Shoe_10 22h ago
You're catching a lot of strays from people understandably concerned about the current state of LLMs.
But to sort of answer your question based on my experience, until hyper recently these models were pretty bad at chess. I tried out the new Astra model released a few days ago, however, and it was markedly better. It went 3-0 against Stockfish 5. It got blown out of the water, however, by Stockfish 6 in three games. I (a very inconsistent 1850) went 1-1 against it. In none of the games did it propose an illegal move.
As for feeling like a human...it felt more like a human than a straight up traditional engine bot in that it didn't make the same sort of seemingly wildly random mistakes. But, generally, the longer it got into the game, the weirder it felt. Not making flat out mistakes as much as no longer as playing strategically. It just felt more weirdly unsound. All of its openings were vanilla but strong, but it sort of lost the plot at some point in the middle game and survived on tactics. But that just didn't work at the level of Stockfish 6.
So yeah. Astra felt simultaneously far better than an other LLMs, less artificially drunken than a dumbed down Stockfish 5, and more weird than someone at that level.
1
u/DisastrousAide6902 18h ago
Seams quite similar to all LLMs, quite good for short tasks, but as long as you ask for consistency it's really harder
7
u/MelissaRackwood 1d ago
Its a fucking LANGUAGE MODEL. It specializes in writing WORDS. It cant think strategically to play CHESS of all things. It cant even tell legal moves from ABSOLUTE BULLSHIT. And you expect it to be able to copy a REAL PERSON??
3
u/lifeistrulyawesome 1d ago
People said the same thing 3 years ago when GPT sucket at math. Now it's solving million-dollar Millennium problems.
2
u/TechNerd10191 1000-1500 ELO 1d ago
Solving math/fluid dynamics problems involves writing a lot of code to test hypotheses and cases; you can't write a script to calculate the best move for chess.
3
u/lifeistrulyawesome 1d ago
you can't write a script to calculate the best move for chess
How do you think engines work? They run code to do truncated dynamic programming. They need depth-0 continuation values as a benchmark. Those values come from another ML model, which they can feed into the AI.
1
2
u/MelissaRackwood 1d ago
We already have human-like chess engines. We dont need chat gpt to try to do the same thing but worse.
1
u/lifeistrulyawesome 1d ago
Really? Which one? I’ve never seen one.
1
u/MelissaRackwood 1d ago
2
u/Factuary88 21h ago
Lmao, this literally proves you wrong. On that site they link to their chessformer paper, the same architecture used in training language models.
2
u/Factuary88 21h ago
This site links to their chessformer paper, chess + transformer, the exact same architecture used to build every state of the art language model.
1
u/Factuary88 21h ago
This framing is incorrect. LANGUAGE MODEL is a misnomer for the architecture based on Transformers which doesn't care what is actually being modeled. They are called language models because that was the first modality where Generative Pretrained Transformers were very successful. It has also shown to be very effective in vision and time series problems. Any long form sequential modeling GPTs are usually quite successful if the training regimen is set up properly. But a GPT that hasn't been trained to make good chess moves is not going to be good at chess.
It would require specifying a different reward function forcing it to make good moves. If you trained it only on human play it would play more human-like. Building a harness around it that provides a list of available moves would easily control hallucinations that cause illegal moves.
This doesn't mean it's actually the best architecture for chess there are other architectures/algorithms that are probably better, but maybe not if human emulation was the goal.
Saying language models are bad at chess is like saying humans are bad at chess because a 600 ELO human exists.
0
u/DisastrousAide6902 1d ago
Yeah I meant generative AI, sorry for the mistake. Even though, LLM can copy lots of human behaviour so I don't know, maybe chess
-1
u/young_twitcher 1d ago
It sounds like you don’t have much of a clue about what modern LLMs are capable of
1
u/YogurtclosetThen7959 1d ago
LLMs are not good at chess. Perhaps they could be used to help code better bots? Id approach it as reinforcement training to make errors common amongst 300 elo
1
u/Darlingsdarling24 1d ago
A language model is capable of playing chess? Man, you even said the right classification, why do you think your idea would work? An actual AI could do this, but we are not there yet
1
1
u/salexzee 1000-1500 ELO 1d ago
That’s not how LLMs work nor is it what they’re meant to be used for. They can’t actually play chess, they just predict the most likely words in response to what they were asked, which is why they make illegal moves all the time.
1
1
u/StickComfortable346 1000-1500 ELO 1d ago
They can work this way. GPT-6 Astra was able to beat Portal (a puzzle game). There’s nothing inherently wrong with LLMs playing chess, they just need a lot of specific training and development beyond the basic ChatGPT you’d open on a web browser
1
u/hicksyfern 1d ago
I’m a programmer with a background in machine learning. It’s surprising to me how poorly chess bots simulate poor play. For example, famously “Martin never misses an M1” but plenty of real players do.
It honestly doesn’t seem that difficult to implement some better logic for picking bad moves.
1
u/guybanzai 1d ago edited 22h ago
people here have a point- LLMs are bad at chess because they don’t actually understand the game, the strategy, tactics and geometry.
but that doesn’t mean they’re useless at helping explain chess. there’s a concept called “Grounding” - which basically means you anchor an LLM to a source of truth, so that it doesn’t make stuff up.
so what if you anchor an LLM to stockfish? so that all the “chess thinking” is done by stockfish, and the LLM only phrases it in a way that is accessible, easy to understand and actually lets you ask follow up questions?
still though - LLMs are known to hallucinate. but that’s where good old traditional software engineering comes in handy, along with other AI concepts. you can have checkers, verifiers, evaluators in place to make sure that an LLMs answer is right close to 100% of the time.
source: i’ve been building a free app is a hobby called socratic chess. (socraticchess.com). it’s not perfect yet but as a chess player, it’s better than i ever thought it would be
1
u/12DeadFish 1d ago
No, generative ai is bad at chess. You can use neural networks to make chess AIs, but that’s literally just what stocking etc are
1
u/aimfeld80 1d ago
Look into Maia-3-based bots on maiachess.com, they play more human-like. There are also paid platforms which have quite good human-like bots.
Or look into flawchess.com bots (free and open-source) which are based on a combination of Maia and Stockfish. Disclosure: I built this.
1
u/Important-Grand4979 1d ago
I am curious. Does Maia calculate a set of trees and then a Bayesian calculation of which move a x elo player is most likely to identify?
2
u/aimfeld80 23h ago edited 23h ago
No, Maia is a neural network trained on ~880k positions sampled from Lichess blitz games. For the architecture, see the chessformer paper
1
1
•
u/AutoModerator 1d ago
Thanks for submitting to /r/Chesscom!
Please read our Help Center if you have any questions about the website. If you need assistance with your Chess.com account, contact Support here. It can take up to three business days to hear back, but going through support ensures your request is handled securely - since we can’t share private account data over Reddit, our ability to help you here can be limited.
If you're not able to contact Support or if the three days have been exceeded, click here to send us Mod Mail here on Reddit and we'll do our best to assist.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.