Cleanup code after dropping ICC support in favor of ICX
The commit removes all uses of ICC's __INTEL_COMPILER macro and other
references to ICC. It also adds ICX info to the compiler command and
fixes two typos in Makefile's help output.
Created by retraining the master net on a dataset composed by:
- adding Leela data from T60 jul-dec 2020, T77 nov 2021, T80 jun-jul 2023
- deduplicating and unminimizing parts of the dataset before interleaving
Trained initially with max epoch 800, then increased near the end of training
twice. First to 960, then 1200. After training, post-processing involved:
- greedy permuting L1 weights with https://github.com/official-stockfish/Stockfish/pull/4620
- greedy 2- and 3- cycle permuting with https://github.com/official-stockfish/Stockfish/pull/4640
In the list of datasets below, periods in the filename represent the sequence of
steps applied to arrive at the particular binpack. For example:
test77-dec2021-16tb7p.filter-v6-dd.min-mar2023.unminimized.binpack
1. test77 dec2021 data rescored with 16 TB of syzygy tablebases during data conversion
2. filtered with csv_filter_v6_dd.py - v6 filtering and deduplication in one step
3. minimized with the original mar2023 implementation of `minimize_binpack` in
the tools branch
4. unminimized by removing all positions with score == 32002 (`VALUE_NONE`)
Michael Chaly [Mon, 11 Sep 2023 12:37:18 +0000 (15:37 +0300)]
Do more futility pruning in qsearch
This patch introduces a third futility pruning heuristic in qsearch. The idea is
that the static exchange evaluation is much worse than the difference between
futility base and alpha. Thus we can assume that the probability of the move
being good enough to beat alpha is low so it can be pruned.
Fixes a bug introduced in 2f2f45f, where with AVX-512 the weights and input to
the last layer were being read out of bounds. Now AVX-512 is only used for the
layers it can be used for. Additional static assertions have been added to
prevent more errors like this in the future.
This is a cleanup PR that prepares the automatic checking of missing or
superfluous #include directives via the include-what-you-use (IWYU) tool
on the CI. Unfortunately, IWYU proposes additional includes for
"namespace std" although we don't need them.
To avoid the problem, the commit removes all "using namespace std"
statements from the code and directly uses the std:: prefix instead.
Alternatively, we could add specific usings (e.g. "using std::string")
foreach used type. Also, a mix of both approaches would be possible.
I decided for the prefix approach because most of the files were already
using the std:: prefixes despite the "using namespace std".
This patch implements the pure materialistic evaluation called simple_eval()
to gain a speed-up during Stockfish search.
We use the so-called lazy evaluation trick: replace the accurate but slow
NNUE network evaluation by the super-fast simple_eval() if the position
seems to be already won (high material advantage). To guard against some
of the most obvious blunders introduced by this idea, this patch uses the
following features which will raise the lazy evaluation threshold in some
situations:
- avoid lazy evals on shuffling branches in the search tree
- avoid lazy evals if the position at root already has a material imbalance
- avoid lazy evals if the search value at root is already winning/losing.
Moreover, we add a small random noise to the simple_eval() term. This idea
(stochastic mobility in the minimax tree) was worth about 200 Elo in the pure
simple_eval() player on Lichess.
Overall, the current implementation in this patch evaluates about 2% of the
leaves in the search tree lazily.
FauziAkram [Fri, 25 Aug 2023 12:42:44 +0000 (15:42 +0300)]
Rename one variable
To enhance code clarity and prevent potential confusion with the
'r' variable assigned to reduction later in the code, this pull
request renames it to 'reductionScale' when we use the same name
in the reduction() function.
Using distinct variable names for separate functions improves code
readability and maintainability.
Disservin [Sat, 26 Aug 2023 07:49:04 +0000 (09:49 +0200)]
Simplify README
The UCI protocol is rather technical and has little value in our README. Instead
it should be explained in our wiki. "Contributing" is moved above "Compiling
Stockfish" to make it more prominent.
Also move the CONTRIBUTING.md into the root directory and include it in the
distributed artifacts/releases.
Disservin [Wed, 23 Aug 2023 17:36:55 +0000 (19:36 +0200)]
Cleanup includes
Reorder a few includes, include "position.h" where it was previously missing
and apply include-what-you-use suggestions. Also make the order of the includes
consistent, in the following way:
1. Related header (for .cpp files)
2. A blank line
3. C/C++ headers
4. A blank line
5. All other header files
Play turbulent when defending, simpler when attacking
This patch decays a little the evaluation (up to a few percent) for
positions which have a large complexity measure (material imbalance,
positional compensations, etc).
This may have nice consequences on the playing style, as it modifies
the search differently for attack and defense, both effects being
desirable:
- to see the effect on positions when Stockfish is defending, let us
suppose for instance that the side to move is Stockfish and the nnue
evaluation on the principal variation is -100 : this patch will decay
positions with an evaluation of -103 (say) to the same level, provided
they have huge material imbalance or huge positional compensation.
In other words, chaotic positions with an evaluation of -103 are now
comparable in our search tree to stable positions with an evaluation
of -100, and chaotic positions with an evaluation of -102 are now
preferred to stable positions with an evaluation of -100.
- the effect on positions when Stockfish is attacking is the opposite.
Let us suppose for instance that the side to move is Stockfish and the
nnue evaluation on the principal variation is +100 : this patch will
decay the evaluation to +97 if the positions on the principal variation
have huge material imbalance or huge positional compensation. In other
words, stable positions with an evaluation of +97 are now comparable
in our search tree to chaotic positions with an evaluation of +100,
and stable positions with an evaluation of +98 are now preferred to
chaotic positions with an evaluation of +100.
So the effect of this small change of evaluation on the playing style
is that Stockfish should now play a little bit more turbulent when
defending, and choose slightly simpler lines when attacking.
Shahin M. Shahin [Sat, 19 Aug 2023 22:15:22 +0000 (01:15 +0300)]
Reduce repetitions branches
Increase reduction on retrying a move we just retreated that falls in a repetition:
if current move can be the same move from previous previous turn then we retreated
that move on the previous turn, this patch increases reduction if retrying that move
results in a repetition.
How to continue from there? Maybe we some variants of this idea could bring Elo too
(only testing the destination square, or triangulations, etc.)
cj5716 [Sat, 12 Aug 2023 06:56:23 +0000 (14:56 +0800)]
Do more full window searches
Remove the value < beta condition for doing full window searches.
As an added bonus the condition for full-window search is now much
more similar to other fail-soft engines.
Squared numbers are never negative, so barring any wraparound there
is no need to clamp to 0. From reading the code, there's no obvious
way to get wraparound, so the entire operation can be simplified
away. Updated original truncated code comments to be sensible.
Verified by running ./stockfish bench 128 1 24 and by the following test:
Matthies [Wed, 16 Aug 2023 09:11:27 +0000 (11:11 +0200)]
Allow compilation on Raspi (for ARMv8)
Current master fails to compile for ARMv8 on Raspi cause gcc (version 10.2.1)
does not like to cast between signed and unsigned vector types. This patch
fixes it by using unsigned vector pointer for ARM to avoid implicite cast.
Disservin [Mon, 14 Aug 2023 11:49:41 +0000 (13:49 +0200)]
Add -funroll-loops to CXXFLAGS
Optimize profiling data accuracy by enabling -funroll-loops during the profile
generation phase, in addition to its default activation by -fprofile-use.
This seems to produce a slightly faster binary, for most compilers.
Besides rebasing I replaced PawnValueMg w/ 126 explicitly to decouple from https://tests.stockfishchess.org/tests/view/64d212de5b17f7c21c0debbb by @peregrineshahin which also passed. #4734
Stéphane Nicolet [Thu, 10 Aug 2023 04:31:48 +0000 (06:31 +0200)]
Simplify SEE pruning for captures
It seems that the current search is smart enough to allow us to remove
(again) the block of code that checks for discovered attacks after the
first pruning condition for captures.
cj5716 [Tue, 8 Aug 2023 09:06:31 +0000 (17:06 +0800)]
Simplify a depth condition
As the negative extension term has sensitive scaling, it would make more sense to allow more negative extension also at lower depth, and not just a region between low and high depth.
a) Add further tests to CI to cover most features. This uncovered a potential race
in case setoption was sent between two searches. As the UCI protocol requires
this sent to be went the engine is not searching, setoption now ensures that
this is the case.
Tomasz Sobczyk [Mon, 7 Aug 2023 11:11:09 +0000 (13:11 +0200)]
Fix Makefile for incorrect nnue file
If an incorrect network file is present at the start of the compilation stage, the
Makefile script now correctly removes it before trying to download a clean version.
Michael Chaly [Sun, 6 Aug 2023 23:32:38 +0000 (02:32 +0300)]
Adjust futility pruning base in qsearch
Current master used value from transposition table there if it existed,
this patch uses minimum between this tt value and the static eval instead
(this thus is closer to the main search function, which uses the static eval).
Based on vondele's deletepsqt branch:
https://github.com/vondele/Stockfish/commit/369f5b051
This huge simplification uses a weighted material differences instead of
the positional piece square tables (psqt) in the semi-classical complexity
calculation. Tuned weights using spsa at 45+0.45 with:
int pawnMult = 100;
int knightMult = 325;
int bishopMult = 350;
int rookMult = 500;
int queenMult = 900;
TUNE(SetRange(0, 200), pawnMult);
TUNE(SetRange(0, 650), knightMult);
TUNE(SetRange(0, 700), bishopMult);
TUNE(SetRange(200, 800), rookMult);
TUNE(SetRange(600, 1200), queenMult);
The values obtained via this tuning session were for a model where
the psqt replacement formula was always from the point of view of White,
even if the side to move was Black. We re-used the same values for an
implementation with a psqt replacement from the point of view of the side
to move, testing the result both on our standard book on positions with
a strong White bias, and an alternate book with positions with a strong
Black bias.
We note that with the patch the last use of the venerable "Score" type
disappears in Stockfish codebase (the Score type was used in classical
evaluation to get a tampered eval interpolating values smoothly from the
early midgame stage to the endgame stage). We leave it to another commit
to clean all occurrences of Score in the code and the comments.
AndrovT [Tue, 1 Aug 2023 12:43:37 +0000 (14:43 +0200)]
Implement AffineTransformSparseInput for armv8
Implements AffineTransformSparseInput layer for the NNUE evaluation
for the armv8 and armv8-dotprod architectures. We measured some nice
speed improvements via 10 runs of our benchmark:
Malus during move ordering for putting pieces en prise
The original idea is the reverse of a previous patch [1] which added bonuses
in our move picker to moves escaping threats. In this patch, in addition to
bonuses for evading threats, we apply penalties to moves moving to threatened
squares.
Further tweaks of that basic idea resulted in this specific version which
further increases the penalty of moves moving to squares threatend depending
on the piece threatening it. So for example a queen moving to a square attacked
by a pawn would receive a larger penalty than a queen moving to square attacked
by a rook.
Also make two get_weight_index() static methods constexpr, for
consistency with the other static get_hash_value() method right above.
Tested for speed by user Torom (thanks).
This patch changes the frequency with which the time is checked, changing
frequency from every 1024 counted nodes to every 512 counted nodes. The
master value was tuned for the old classical eval, the patch takes the
roughly 2x slowdown in nps with SFNNUEv7 into account. This could reduce
a bit the losses on time on fishtest, but they are probably unrelated.
Michael Chaly [Wed, 19 Jul 2023 08:58:02 +0000 (11:58 +0300)]
Do more futility pruning for cutNodes that are not in TT
This is somewhat similar to IIR for cutnodes but instead of reducing
depth for cutnodes that don't have tt move we reduce margin multiplier
in futility pruning for cutnodes that are not in TT.
Explicitly describe the architecture as deprecated,
it remains available as its current alias x86-64-sse41-popcnt
CPUs that support just this instruction set are now years old,
any few years old Intel or AMD CPU supports x86-64-avx2. However,
naming things 'modern' doesn't age well, so instead use explicit names.
Adjust CI accordingly. Wiki, fishtest, downloader done as well.
use a fixed compiler on Linux and Windows (right now gcc 11).
build avxvnni on Windows (Linux needs updated core utils)
build x86-32 on Linux (Windows needs other mingw)
fix a Makefile issue where a failed PGOBENCH would not stop the build
reuse the WINE_PATH for SDE as we do for QEMU
use WINE_PATH variable also for the signature
verify the bench for each of the binaries
do not build x86-64-avx2 on macos
use intel's Software Development Emulator (SDE) in the actions that build the binaries.
This allows for building on Windows and Linux binaries for
- x86-64-avx512
- x86-64-vnni256
- x86-64-vnni512
(x86-64-avxvnni needs more recent gcc in the actions)
since the introduction of NNUE (first released with Stockfish 12), we
have maintained the classical evaluation as part of SF in frozen form.
The idea that this code could lead to further inputs to the NN or
search did not materialize. Now, after five releases, this PR removes
the classical evaluation from SF. Even though this evaluation is
probably the best of its class, it has become unimportant for the
engine's strength, and there is little need to maintain this
code (roughly 25% of SF) going forward, or to expend resources on
trying to improve its integration in the NNUE eval.
Indeed, it had still a very limited use in the current SF, namely
for the evaluation of positions that are nearly decided based on
material difference, where the speed of the classical evaluation
outweights its inaccuracies. This impact on strength is small,
roughly 2Elo, and probably decreasing in importance as the TC grows.
Potentially, removal of this code could lead to the development of
techniques to have faster, but less accurate NN evaluation,
for certain positions.
peregrineshahin [Tue, 27 Jun 2023 03:07:20 +0000 (06:07 +0300)]
Fix pruning to (in TB loss) in Null move pruning.
Current logic can apply Null move pruning
on a dead-lost position returning an unproven loss
(i.e. in TB loss score or mated in losing score) on nonPv nodes.
on a default bench, this can be observed by adding this debugging line:
```
if (nullValue >= beta)
{
// Do not return unproven mate or TB scores
nullValue = std::min(nullValue, VALUE_TB_WIN_IN_MAX_PLY-1);
dbg_hit_on(nullValue <= VALUE_TB_LOSS_IN_MAX_PLY); // Hit #0: Total 73983 Hits 1 Hit Rate (%) 0.00135166
if (thisThread->nmpMinPly || depth < 14)
return nullValue;
```
This fixes this very rare issue (happens at ~0.00135166% of the time) by
eliminating the need to try Null Move Pruning with dead-lost positions
and leaving it to be determined by a normal searching flow.
The previous try to fix was not as safe enough because it was capping
the returned value to (out of TB range) thus reviving the dead-lost position
based on an artificial clamp (i.e. the in TB score/mate score can be lost on that nonPv node):
https://tests.stockfishchess.org/tests/view/649756d5dc7002ce609cd794
In CI, it is typical for the process to halt immediately when an error
is encountered. However, with our `shell: bash {0}` configuration,
the process continues despite errors for posix shells.
This commit updates the behavior of posix and msys2 shells to ensure
consistency in terms of pipeline exit codes and stop conditions.
We adopt the most appropriate default behavior as recommended
by the GitHub documentation.
Update the code that searches for the bench value in the git log:
- to be compatible with the new shell settings
- to retry the value from the first line that contains
only the template and spaces/tabs/newlines
Linmiao Xu [Sun, 25 Jun 2023 21:44:28 +0000 (17:44 -0400)]
Update NNUE architecture to SFNNv7 with larger L1 size of 2048
Creating this net involved:
- a 5-step training process from scratch
- greedy permuting L1 weights with https://github.com/official-stockfish/Stockfish/pull/4620
- leb128 compression with https://github.com/glinscott/nnue-pytorch/pull/251
- greedy 2- and 3- cycle permuting with https://github.com/official-stockfish/Stockfish/pull/4640
The 5 training steps were:
1. 400 epochs, lambda 1.0, lr 9.75e-4
UHOx2-wIsRight-multinet-dfrc-n5000-largeGensfen-d9.binpack (178G)
nodes5000pv2_UHO.binpack
data_pv-2_diff-100_nodes-5000.binpack
wrongIsRight_nodes5000pv2.binpack
multinet_pv-2_diff-100_nodes-5000.binpack
dfrc_n5000.binpack
large_gensfen_multipvdiff_100_d9.binpack
ep399 chosen as start model for step2
2. 800 epochs, end-lambda 0.75, skip 16
LeelaFarseer-T78juntoaugT79marT80dec.binpack (141G)
T60T70wIsRightFarseerT60T74T75T76.binpack
test78-junjulaug2022-16tb7p.no-db.min.binpack
test79-mar2022-16tb7p.no-db.min.binpack
test80-dec2022-16tb7p.no-db.min.binpack
ep559 chosen as start model for step3
3. 800 epochs, end-lambda 0.725, skip 20
leela96-dfrc99-v2-T80dectofeb-sk20-mar-v6-T77decT78janfebT79apr.binpack (223G)
leela96-filt-v2.min.binpack
dfrc99-16tb7p-eval-filt-v2.min.binpack
test80-dec2022-16tb7p-filter-v6-sk20.min-mar2023.binpack
test80-jan2023-16tb7p-filter-v6-sk20.min-mar2023.binpack
test80-feb2023-16tb7p-filter-v6-sk20.min-mar2023.binpack
test80-mar2023-2tb7p-filter-v6.min.binpack
test77-dec2021-16tb7p.no-db.min.binpack
test78-janfeb2022-16tb7p.no-db.min.binpack
test79-apr2022-16tb7p.no-db.min.binpack
ep499 chosen as start model for step4
4. 800 epochs, end-lambda 0.7, skip 24 0dd1cebea57 dataset https://github.com/official-stockfish/Stockfish/pull/4606
ep599 chosen as start model for step5
5. 800 epochs, end-lambda 0.7, skip 28
same dataset as step4
ep619 became nn-1b951f8b449d.nnue
SF training data components for the step1 dataset:
https://drive.google.com/drive/folders/1yLCEmioC3Xx9KQr4T7uB6GnLm5icAYGU
Leela training data for steps 2-5 can be found at:
https://robotmoon.com/nnue-training-data/
Due to larger L1 size and slower inference, the speed penalty loses elo
at STC. Measurements from 100 bench runs at depth 13 with x86-64-modern
on Intel Core i5-1038NG7 2.00GHz:
cj5716 [Mon, 26 Jun 2023 11:40:22 +0000 (19:40 +0800)]
Negative extension on cutNodes based on depth
This patch was inspired by candirufish original attempt at negative extensions
here that failed yellow: https://tests.stockfishchess.org/tests/view/6486529065ffe077ca124f32
I tested some variations of the idea and tuned a depth condition for
a modified version of it here https://tests.stockfishchess.org/tests/view/648db80a91c58631ce31fe00
after noticing abnormal scaling (ie many passed STC but not LTC)
After some small tweaks I got the final version here
Activation data taken from https://drive.google.com/drive/folders/1Ec9YuuRx4N03GPnVPoQOW70eucOKngQe?usp=sharing
Permutation found using https://github.com/Ergodice/nnue-pytorch/blob/836387a0e5e690431d404158c46648710f13904d/ftperm.py
See also https://github.com/glinscott/nnue-pytorch/pull/254
The algorithm greedily selects 2- and 3-cycles that can be permuted to increase the number of runs of zeroes. The percent of zero runs from the master net increased from 68.46 to 70.11 from 2-cycles and only increased to 70.32 when considering 3-cycles. Interestingly, allowing both halves of L1 to intermix when creating zero runs can give another 0.5% zero-run density increase with this method.
Measured speedup:
```
CPU: 16 x AMD Ryzen 9 3950X 16-Core Processor
Result of 50 runs
base (./stockfish.master ) = 1561556 +/- 5439
test (./stockfish.patch ) = 1575788 +/- 5427
diff = +14231 +/- 2636
A new major release of Stockfish is now available at
https://stockfishchess.org/download/
*Quality of chess play*
Stockfish continues to demonstrate its ability to discover superior moves
with remarkable speed. In self-play against Stockfish 15, this new
release gains up to 50 Elo[1] and wins up to 12 times more game pairs[2]
than it loses. In major chess engine tournaments, Stockfish reliably tops
the rankings[3] winning the TCEC season 24 Superfinal, Swiss, Fischer
Random, and Double Random Chess tournaments and the CCC 19 Bullet,
20 Blitz, and 20 Rapid competitions. Leela Chess Zero[4] was the
challenger in most finals, putting top-engine chess now firmly in the
hands of teams embracing free and open-source software.
*Progress made*
This updated version of Stockfish introduces several enhancements,
including an upgraded neural net architecture (SFNNv6)[5], improved
implementation, and refined parameterization. The ongoing utilization
of Leela’s data combined with a novel inference approach exploiting
sparsity[6], and network compression[7] ensure a speedy evaluation and
modest binary sizes while allowing for more weights and higher accuracy.
The search has undergone more optimization, leading to improved
performance, particularly in longer analyses[8]. Additionally,
the Fishtest framework has been improved and is now able to run the
tests needed to validate new ideas with 10000s of CPU cores.
*Usability improvements*
Stockfish now comes with documentation, found in the wiki folder when
downloading it or on GitHub[9]. Additionally, Stockfish now includes
a clear and consistent forced tablebase win score, displaying a value
of 200 minus the number of plies required to reach a tablebase win[10].
Furthermore, the UCI_Elo option, to reduce its strength, has been
calibrated[11]. It is worth noting that the evaluation system remains
consistent with Stockfish 15.1[12], maintaining the choice that 100cp
means a 50% chance of winning the game against an equal opponent[13].
Finally, binaries of our latest development version are now provided
continuously as pre-releases on GitHub making it easier for
enthusiasts to download the latest and strongest version of
the program[14], we thank Roman Korba for having provided a similar
service for a long time.
*Thank you*
The success of the Stockfish project relies on the vibrant community
of passionate enthusiasts (we appreciate each and every one of you!)
who generously contribute their knowledge, time, and resources.
Together, this dedicated community works towards the common goal of
developing a powerful, freely accessible, and open-source chess engine.
We invite all chess enthusiasts to join the Fishtest testing framework
and contribute to the project[15].
projects / stockfish / log