Muzhen Gaming [Sat, 25 Nov 2023 14:16:56 +0000 (22:16 +0800)]
Simplify move history reduction
Recent VLTC search tuning has suggested that the depth limit can be increased
by a lot. This patch simplifies away the depth-based bonus from statScore
reduction, making the divisor a constant.
We use following line to clamp the search depth in some range:
Depth d = std::clamp(newDepth - r, 1, newDepth + 1);
Through negative extension its possible that the maximum value becomes smaller than the minimum value but then the behavior is undefined (see https://en.cppreference.com/w/cpp/algorithm/clamp). So replace this line with a safe implementation.
Remark:
We have in recent master already one line where up to 3 negative extensions are possible which could trigger this undefined behavior but this can only be happen for completed depth > 24 so its not discovered by our default bench. Recent negative extension tests by @fauzi shows then this undefined behavior with wrong bench numbers.
- updates the SDE action to v2.2
- removes the linux x86-32 builds, which were almost unused,
and the build process under SDE started failing recently,
possibly related to glibc update (The futex facility returned an unexpected error code.)
Double weight of pawn history for quiet move ordering.
I measured on my 1000 position bench the average additional added pawn history per depth.
This shows on average negative value with even smaller values with increaing depth.
A linear regression against depth get following formula:
-1960 - 130 * depth
For compensation add this to the used sort limit to maintain roughly the same proportion of sorted quiet moves.
Remarks:
1. using no compensation failed here https://tests.stockfishchess.org/tests/view/6547664f136acbc5735265f0
2. using only the compensation failed at LTC:
passed STC: https://tests.stockfishchess.org/tests/view/65477457136acbc5735266f8
failed LTC: https://tests.stockfishchess.org/tests/view/65487fc8136acbc573527d1c
FauziAkram [Sat, 4 Nov 2023 14:19:08 +0000 (17:19 +0300)]
Tweaking the futility pruning formula
Huge credit goes also to candirufish,
as the idea was first tried by him, and then tuned by me at multiple phases.
Tweaking the futility pruning formula to be a bit more selective about when pruning is applied.
Adjust the value added to the static eval based on the bestValue relative to ss->staticEval. If bestValue is significantly lower, we add a larger value.
The recently commit 'Rewarding Quiet Moves that Enable Razoring' add a history update if razoring. But its contains also many tuned values all over the search. Following tests shows that the tuned values and not the added history update is responsible for the elo gain. So remove later.
FauziAkram [Fri, 27 Oct 2023 10:28:55 +0000 (13:28 +0300)]
Rewarding Quiet Moves that Enable Razoring
The main idea of the patch comes from @peregrineshahin :
https://tests.stockfishchess.org/tests/view/65205363ac57711436728781
Another small idea (tuning) comes from @Vizvezdenec
https://tests.stockfishchess.org/tests/view/652e071ade6d262d08d318f4 And a long
phases of tuning and tests was done by me in order to make the patch be able to
pass both tests.
The idea, as mentioned by Peregrine is that in our standard code, if no best
move found after searching all moves, we give a bonus to the previous move that
caused the fail high. So in razoring we assume no bestmove will be found so we
might as well do the same.
Michael Chaly [Fri, 27 Oct 2023 15:19:31 +0000 (17:19 +0200)]
Introduce pawn structure based history
Original idea by Seer chess engine https://github.com/connormcmonigle/seer-nnue,
coding done by @Disservin, code refactoring done by @locutus2 to match the style
of other histories.
This patch introduces pawn structure based history, which assings moves values
based on last digits of pawn structure hash and piece type of moved piece and
landing square of the move. Idea is that good places for pieces are quite often
determined by pawn structure of position. Used in 3 different places
- sorting of quiet moves, sorting of quiet check evasions and in history based
pruning in search.
Muzhen Gaming [Sat, 21 Oct 2023 09:01:45 +0000 (17:01 +0800)]
Time management improvements
1. Tune time management parameters.
2. Scale the optimum time and maximum time parameters based on the amount of
time left, using a logarithmic scale.
Many acknowledgements to @FauziAkram for tuning the parameters and for the
original idea (see
https://tests.stockfishchess.org/tests/view/652f0356de6d262d08d333c5).
Disservin [Sun, 22 Oct 2023 18:20:53 +0000 (20:20 +0200)]
remove blank line between function and it's description
- remove the blank line between the declaration of the function and it's
comment, leads to better IDE support when hovering over a function to see it's
description
- remove the unnecessary duplication of the function name in the functions
description
- slightly refactored code for lsb, msb in bitboard.h There are still a few
things we can be improved later on, move the description of a function where
it was declared (instead of implemented) and add descriptions to functions
which are behind macros ifdefs
Disservin [Sun, 22 Oct 2023 14:43:33 +0000 (16:43 +0200)]
use expanded variables for shell commands
Performance improvement for the shell commands in the Makefile.
By using expanded variables, the shell commands are only
evaluated once, instead of every time they are used.
Taras Vuk [Sun, 22 Oct 2023 06:36:43 +0000 (08:36 +0200)]
Simplify futilityBase formula
This patch replaces std::min(ss->staticEval, bestValue) with ss->staticEval in the futilityBase formula.
Original idea by Vizvezdenec: https://tests.stockfishchess.org/tests/view/64ce66795b17f7c21c0d85f3
Disservin [Sat, 21 Oct 2023 09:40:56 +0000 (11:40 +0200)]
add clang-format
This introduces clang-format to enforce a consistent code style for Stockfish.
Having a documented and consistent style across the code will make contributing easier
for new developers, and will make larger changes to the codebase easier to make.
To facilitate formatting, this PR includes a Makefile target (`make format`) to format the code,
this requires clang-format (version 17 currently) to be installed locally.
Installing clang-format is straightforward on most OS and distros
(e.g. with https://apt.llvm.org/, brew install clang-format, etc), as this is part of quite commonly
used suite of tools and compilers (llvm / clang).
Additionally, a CI action is present that will verify if the code requires formatting,
and comment on the PR as needed. Initially, correct formatting is not required, it will be
done by maintainers as part of the merge or in later commits, but obviously this is encouraged.
This patch is a simplification and a fix to dealing with null moves scores that returns proven mates or TB scores by preventing 'null move pruning' if the nullvalue is in that range.
Current solution downgrades nullValues on the non-PV node but the value can be used in a transposed PV-node to the same position afterwards (Triangulation), the later is prone to propagate a wrong score (96.05) to root that will not be refuted unless we search further.
Score of (96.05) can be obtained be two methods,
maxim static-eval returned on Pv update (mostly qSearch)
this downgrade (clamp) in NMP
and theoretically can happen with or without TBs but the second scenario is more dangerous than the first.
This fixes the reproducible case in very common scenarios with TBs as shown in the debugging at discord.
Michael Chaly [Sat, 14 Oct 2023 14:41:41 +0000 (17:41 +0300)]
Use more continuation histories.
This patch allows stats updates and movepicker bonuses for continuation history 3 plies deep - so counter counter move.
Updates and movepicker usage are done with 1/4 multiplier compared to other histories.
Simplify collection of bad moves for history updates.
1. collect only the first 32 moves searched and ignore the rest. So late bad moves get no further negative history updates.
2. collect now for quiet moves also at most 32 bad moves
mstembera [Sun, 1 Oct 2023 06:12:02 +0000 (23:12 -0700)]
Optimize the most common update accumalator cases w/o tiling
In the most common case where we only update a single state
it's faster to not use temporary accumulation registers and tiling.
(Also includes a couple of small cleanups.)
A simpler version
https://tests.stockfishchess.org/tests/view/65190dfacff46e538ee00155
also passed but this version is stronger still
https://tests.stockfishchess.org/tests/view/6519b95fcff46e538ee00fa2
This is a later epoch from the same experiment that led to the previous
master net. In training stage 6, max-epoch was raised to 1,200 near the
end of the first 1,000 epochs.
For more details, see https://github.com/official-stockfish/Stockfish/pull/4795
Local elo at 25k nodes per move (vs. L1-2048 nn-1ee1aba5ed4c.nnue)
ep1079 : 15.6 +/- 1.2
Linmiao Xu [Fri, 12 May 2023 22:07:20 +0000 (18:07 -0400)]
Update NNUE architecture to SFNNv8: L1-2560 nn-ac1dbea57aa3.nnue
Creating this net involved:
- a 6-stage training process from scratch. The datasets used in stages 1-5 were fully minimized.
- permuting L1 weights with https://github.com/official-stockfish/nnue-pytorch/pull/254
A strong epoch after each training stage was chosen for the next. The 6 stages were:
```
1. 400 epochs, lambda 1.0, default LR and gamma
UHOx2-wIsRight-multinet-dfrc-n5000 (135G)
nodes5000pv2_UHO.binpack
data_pv-2_diff-100_nodes-5000.binpack
wrongIsRight_nodes5000pv2.binpack
multinet_pv-2_diff-100_nodes-5000.binpack
dfrc_n5000.binpack
5. 960 epochs, end-lambda 0.7, LR 4.375e-4, gamma 0.995, skip 28
Increased max-epoch to 960 near the end of the first 800 epochs 5af11540bbfe dataset: https://github.com/official-stockfish/Stockfish/pull/4635
6. 1000 epochs, end-lambda 0.7, LR 4.375e-4, gamma 0.995, skip 28
Increased max-epoch to 1000 near the end of the first 800 epochs 1ee1aba5ed dataset: https://github.com/official-stockfish/Stockfish/pull/4782
```
After removing classic evaluation VALUE_KNOWN_WIN is not anymore returned explicit evaluation. So remove and replace it with VALUE_TB_WIN_IN_MAX_PLY.
Measurement on my big bench (bench 16 1 16 pos1000.fen) verifies that at least with current net the calculated evaluation lies always in the open interval (-VALUE_KNOWN_WIN, VALUE_KNOWN_WIN).
So i consider this a non-functional change. But to be safe i tested this also at LTC as requested by Stephane Nicolet.
The commit adds a CI workflow that uses the included-what-you-use (IWYU)
tool to check for missing or superfluous includes in .cpp files and
their corresponding .h files. This means that some .h files (especially
in the nnue folder) are not checked yet.
The CI setup looks like this:
- We build IWYU from source to include some yet unreleased fixes.
This IWYU version targets LLVM 17. Thus, we get the latest release
candidate of LLVM 17 from LLVM's nightly packages.
- The Makefile now has an analyze target that just build the object
files (without linking)
- The CI uses the analyze target with the IWYU tool as compiler to
analyze the compiled .cpp file and its corresponding .h file.
- If IWYU suggests a change the build fails (-Xiwyu --error).
- To avoid false positives we use LLVM's libc++ as standard library
- We have a custom mappings file that adds some mappings that are
missing in IWYU's default mappings
We also had to add one IWYU pragma to prevent a false positive in
movegen.h.
uses a posix compatible script to find the native arch.
(based on ppigazzini's https://github.com/ppigazzini/stockfish-downloader )
use that native arch by default, no changes if ARCH is specified explicitly.
SF can now be compiled in an optimal way simply using
Now that qsearch has its own repetition detection we can flip the order of lines and remove the guard of depth < 0 which is not needed after reordering (i.e. it was there to prevent checking repetition again at depth ==0).
Cleanup code after dropping ICC support in favor of ICX
The commit removes all uses of ICC's __INTEL_COMPILER macro and other
references to ICC. It also adds ICX info to the compiler command and
fixes two typos in Makefile's help output.
Created by retraining the master net on a dataset composed by:
- adding Leela data from T60 jul-dec 2020, T77 nov 2021, T80 jun-jul 2023
- deduplicating and unminimizing parts of the dataset before interleaving
Trained initially with max epoch 800, then increased near the end of training
twice. First to 960, then 1200. After training, post-processing involved:
- greedy permuting L1 weights with https://github.com/official-stockfish/Stockfish/pull/4620
- greedy 2- and 3- cycle permuting with https://github.com/official-stockfish/Stockfish/pull/4640
In the list of datasets below, periods in the filename represent the sequence of
steps applied to arrive at the particular binpack. For example:
test77-dec2021-16tb7p.filter-v6-dd.min-mar2023.unminimized.binpack
1. test77 dec2021 data rescored with 16 TB of syzygy tablebases during data conversion
2. filtered with csv_filter_v6_dd.py - v6 filtering and deduplication in one step
3. minimized with the original mar2023 implementation of `minimize_binpack` in
the tools branch
4. unminimized by removing all positions with score == 32002 (`VALUE_NONE`)
Michael Chaly [Mon, 11 Sep 2023 12:37:18 +0000 (15:37 +0300)]
Do more futility pruning in qsearch
This patch introduces a third futility pruning heuristic in qsearch. The idea is
that the static exchange evaluation is much worse than the difference between
futility base and alpha. Thus we can assume that the probability of the move
being good enough to beat alpha is low so it can be pruned.
Fixes a bug introduced in 2f2f45f, where with AVX-512 the weights and input to
the last layer were being read out of bounds. Now AVX-512 is only used for the
layers it can be used for. Additional static assertions have been added to
prevent more errors like this in the future.
This is a cleanup PR that prepares the automatic checking of missing or
superfluous #include directives via the include-what-you-use (IWYU) tool
on the CI. Unfortunately, IWYU proposes additional includes for
"namespace std" although we don't need them.
To avoid the problem, the commit removes all "using namespace std"
statements from the code and directly uses the std:: prefix instead.
Alternatively, we could add specific usings (e.g. "using std::string")
foreach used type. Also, a mix of both approaches would be possible.
I decided for the prefix approach because most of the files were already
using the std:: prefixes despite the "using namespace std".
This patch implements the pure materialistic evaluation called simple_eval()
to gain a speed-up during Stockfish search.
We use the so-called lazy evaluation trick: replace the accurate but slow
NNUE network evaluation by the super-fast simple_eval() if the position
seems to be already won (high material advantage). To guard against some
of the most obvious blunders introduced by this idea, this patch uses the
following features which will raise the lazy evaluation threshold in some
situations:
- avoid lazy evals on shuffling branches in the search tree
- avoid lazy evals if the position at root already has a material imbalance
- avoid lazy evals if the search value at root is already winning/losing.
Moreover, we add a small random noise to the simple_eval() term. This idea
(stochastic mobility in the minimax tree) was worth about 200 Elo in the pure
simple_eval() player on Lichess.
Overall, the current implementation in this patch evaluates about 2% of the
leaves in the search tree lazily.
FauziAkram [Fri, 25 Aug 2023 12:42:44 +0000 (15:42 +0300)]
Rename one variable
To enhance code clarity and prevent potential confusion with the
'r' variable assigned to reduction later in the code, this pull
request renames it to 'reductionScale' when we use the same name
in the reduction() function.
Using distinct variable names for separate functions improves code
readability and maintainability.
Disservin [Sat, 26 Aug 2023 07:49:04 +0000 (09:49 +0200)]
Simplify README
The UCI protocol is rather technical and has little value in our README. Instead
it should be explained in our wiki. "Contributing" is moved above "Compiling
Stockfish" to make it more prominent.
Also move the CONTRIBUTING.md into the root directory and include it in the
distributed artifacts/releases.
Disservin [Wed, 23 Aug 2023 17:36:55 +0000 (19:36 +0200)]
Cleanup includes
Reorder a few includes, include "position.h" where it was previously missing
and apply include-what-you-use suggestions. Also make the order of the includes
consistent, in the following way:
1. Related header (for .cpp files)
2. A blank line
3. C/C++ headers
4. A blank line
5. All other header files
Play turbulent when defending, simpler when attacking
This patch decays a little the evaluation (up to a few percent) for
positions which have a large complexity measure (material imbalance,
positional compensations, etc).
This may have nice consequences on the playing style, as it modifies
the search differently for attack and defense, both effects being
desirable:
- to see the effect on positions when Stockfish is defending, let us
suppose for instance that the side to move is Stockfish and the nnue
evaluation on the principal variation is -100 : this patch will decay
positions with an evaluation of -103 (say) to the same level, provided
they have huge material imbalance or huge positional compensation.
In other words, chaotic positions with an evaluation of -103 are now
comparable in our search tree to stable positions with an evaluation
of -100, and chaotic positions with an evaluation of -102 are now
preferred to stable positions with an evaluation of -100.
- the effect on positions when Stockfish is attacking is the opposite.
Let us suppose for instance that the side to move is Stockfish and the
nnue evaluation on the principal variation is +100 : this patch will
decay the evaluation to +97 if the positions on the principal variation
have huge material imbalance or huge positional compensation. In other
words, stable positions with an evaluation of +97 are now comparable
in our search tree to chaotic positions with an evaluation of +100,
and stable positions with an evaluation of +98 are now preferred to
chaotic positions with an evaluation of +100.
So the effect of this small change of evaluation on the playing style
is that Stockfish should now play a little bit more turbulent when
defending, and choose slightly simpler lines when attacking.
Shahin M. Shahin [Sat, 19 Aug 2023 22:15:22 +0000 (01:15 +0300)]
Reduce repetitions branches
Increase reduction on retrying a move we just retreated that falls in a repetition:
if current move can be the same move from previous previous turn then we retreated
that move on the previous turn, this patch increases reduction if retrying that move
results in a repetition.
How to continue from there? Maybe we some variants of this idea could bring Elo too
(only testing the destination square, or triangulations, etc.)
cj5716 [Sat, 12 Aug 2023 06:56:23 +0000 (14:56 +0800)]
Do more full window searches
Remove the value < beta condition for doing full window searches.
As an added bonus the condition for full-window search is now much
more similar to other fail-soft engines.
Squared numbers are never negative, so barring any wraparound there
is no need to clamp to 0. From reading the code, there's no obvious
way to get wraparound, so the entire operation can be simplified
away. Updated original truncated code comments to be sensible.
Verified by running ./stockfish bench 128 1 24 and by the following test:
Matthies [Wed, 16 Aug 2023 09:11:27 +0000 (11:11 +0200)]
Allow compilation on Raspi (for ARMv8)
Current master fails to compile for ARMv8 on Raspi cause gcc (version 10.2.1)
does not like to cast between signed and unsigned vector types. This patch
fixes it by using unsigned vector pointer for ARM to avoid implicite cast.