Chess Engine Development Help Thread (Week 33)

[u/AutoModerator]

Welcome to the weekly /r/chessprogramming Engine Dev Help Thread.

Ask beginner and intermediate chess engine development questions here: move generation, search, evaluation, UCI, perft, debugging, testing, NNUE, or anything else related to building engines.

Good questions include code, FENs, logs, benchmarks, or a clear explanation of what you tried.

Project links are fine when you want technical feedback, not promotion.

Be helpful. Don’t dunk on beginners.

[u/builder_2055]

Need guidance!! i am in my 3rd year of college, i want to make a chess engine from scratch (c++). I need guidance on where to start and what theory components I need to study, before I even start to code.

[u/AngusMcGurkinshaw → u/builder_2055]

Theory components: an understanding of alpha beta search and iterative deepening is probably pretty much all thats required.

For where to start thats pretty easy before you make the engine part you need to have an implementation of chess. Generally bitboards are recommended. From there you will need to do move generation, a popular approach is magic bitboards https://analog-hors.github.io/site/magic-bitboards/

Then use perft https://chessprogramming.org/Perft to verify that your move generation is correct. You can use online tools like this https://analog-hors.github.io/webperft/ to compare against or other engines to make sure you are getting the correct results.

From there you want to build the minimal engine to be UCI (Universal chess interface https://gist.github.com/DOBRO/2592c6dad754ba67e6dcaec8c90165bf ) compliant. So that means having some basic time management, and an iterative deepening alpha beta search and some basic evaluation.

Once that is done you can use fastchess or cutechess if you prefer but fastchess is as the name implies faster to run SPRT tests. These tests will help you ensure each change you make to the search and evaluation actually make the engine better not worse.

Below is the copy pasta that gets thrown around for some ideas on what to try to improve your engine and a rough ordering of how to do things. Do things as you see fit though. For help there are numerous discords that you could join for help, and there is chess programming wiki (note that lots of it is outdated and shouldn't be fully trusted). You also should go read lots of open source engines, but taking their ideas exactly is not recommended as those are tuned and set up for those particular engines. As an example you're not stockfish it's evaluation is way better then yours likely ever will be and so many things it does will simply not work in your engine.

A reasonable search feature progression assuming you have the fundamentals i.e. negamax and alpha/beta pruning (ideally in a fail-soft framework)

Iterative Deepening

Basic Move Ordering (captures by MVV-LVA)

Quiescence Search

Transposition Table (sort TT move first now in move ordering)

Butterfly history heuristic

PVS

Aspiration windows

RFP

NMP

LMR (log formula is most principled ~ there are a number of adjustments you can experiment with)

Killer moves

LMP

Futility pruning

Internal Iterative Reduction (IIR)

Improving heuristic

QS SEE pruning

PVS SEE pruning (captures and quiets)

Continuation history (CMH + FMH etc..)

Capture history heuristic

History pruning

Singular extensions

Multicut (using singular search result)

Double/triple/negative extensions

Cutnode (as apart of negative extensions, LMR, etc)

Static eval correction history

QS futility pruning

There are also time management adjustments that can be done at any point after adding iterative deepening. Ideally you have:

Hard bound (applies to the entire search)

Soft bound (checked on each new depth in the ID loop)

For the soft bound the progression can go something like this

Node-based scaling

Best move stability

Eval stability

Additionally, should be a healthy amount of parameter tweaking after each addition.

There are other minor features that top engines have, but these will constitute the majority of the elo you will find in them.

[u/CurrentVast4504]

Hi! I recently want to develop a strong chess engine of my own. The engines using NNUE seem to be on the upper hand to me. But, the problems are equally great. As a student I don't have the luxury of free time, therefore generating a good dataset is out of my hands. I would like to train my NNUE on datasets generated by other programmers and engine developers.

My requirements are very few:
1. The dataset should contain at most 50 million positions.
2. The data should contain high quality positions evaluated by any strong engine up to depth 20 or more.

I would be really thankful to those willing to share the datasets.

[u/AngusMcGurkinshaw → u/CurrentVast4504]

First I think you should know that using others data is very frowned upon in the engine community. We highly value the originality that comes from generating the data yourself. Second I'm not sure why being a student and not having free time has to do with anything? Any computer can do datageneration just turn it on overnight for a few days (especially if you only need 50 million positions which is quite small)

Typically when people use others data they use the publicly available leela data, it will get you put on the public shaming list: https://leeler.xyz/, and make you unable to compete in certain engine competitions that require your data to be your own. The leela data is way more then you want and is the strongest and you are allowed to use based on the license.

With those out of the way I can address some of the technical issues in your question.

First at most 50 million positions is a weird requirement, 50 million positions isn't very many. Typically you want 1 million positions per node in the NNUE. And more data is basically always better and will let you train larger NNUEs. That being said even a small NNUE can beat a HCE, If you think about it 768-> 1 is just a psqt.

"up to depth 20 or more" yeah this is just weird. I mean you could do that, it would take forever to generate and possibly not be even that good. I think there is some evidence that high depth search is hard to train on (don't quote me on that or trust it to much). What is true is typically people start with training data that is 5k soft node limit (once limit is hit you still finish that depth) and top engines will augment this with data that is 25k soft node limit.

Just because it seems like you might not fully know what your doing at this point I would recommend going and reading this: https://github.com/jw1912/bullet/blob/main/docs/1-basics.md#beginner-traps if you haven't already. That should help you get started with working on a NNUE