mstembera [Sun, 16 Aug 2020 22:23:50 +0000 (15:23 -0700)]
Fallback to NNUE
If the classical eval ends up much smaller than estimated fall back to NNUE.
Also use multiply instead of divide for the threshold comparison for smoother transitions without rounding.
notruck [Sun, 16 Aug 2020 15:59:13 +0000 (08:59 -0700)]
Support building for Android using NDK
The easiest way to use the NDK in conjunction with this Makefile (tested on linux-x86_64):
1. Download the latest NDK (r21d) from Google from https://developer.android.com/ndk/downloads
2. Place and unzip the NDK in $HOME/ndk folder
3. Export the path variable e.g., `export PATH=$PATH:$HOME/ndk/android-ndk-r21d/toolchains/llvm/prebuilt/linux-x86_64/bin`
4. cd to your Stockfish/src dir
5. Issue `make -j ARCH=armv8 COMP=ndk build` (use `ARCH=armv7` or `ARCH=armv7-neon` for older CPUs)
6. Optionally `make -j ARCH=armv8 COMP=ndk strip`
7. That's all. Enjoy!
Improves support from Raspberry Pi (incomplete?) and compiling on arm in general
Support is still fragile as we're missing CI on these targets. Nevertheless tested with:
```bash
# build crosses from ubuntu 20.04 on x86 to various arch/OS combos
# tested with suitable packages installed
# (build-essentials, mingw-w64, g++-arm-linux-gnueabihf, NDK (r21d) from google)
# cross to Android
export PATH=$HOME/ndk/android-ndk-r21d/toolchains/llvm/prebuilt/linux-x86_64/bin:$PATH
make clean && make -j build ARCH=armv7 COMP=ndk && make -j build ARCH=armv7 COMP=ndk strip
make clean && make -j build ARCH=armv7-neon COMP=ndk && make -j build ARCH=armv7-neon COMP=ndk strip
make clean && make -j build ARCH=armv8 COMP=ndk && make -j build ARCH=armv8 COMP=ndk strip
# cross to Raspberry Pi
make clean && make -j build ARCH=armv7 COMP=gcc COMPILER=arm-linux-gnueabihf-g++
make clean && make -j build ARCH=armv7-neon COMP=gcc COMPILER=arm-linux-gnueabihf-g++
# cross to Windows
make clean && make -j build ARCH=x86-64-modern COMP=mingw
```
syzygy1 [Sat, 15 Aug 2020 14:50:39 +0000 (16:50 +0200)]
Assume network file is in little-endian byte order
This patch fixes the byte order when reading 16- and 32-bit values from the network file on a big-endian machine.
Bytes are ordered in read_le() using unsigned arithmetic, which doesn't need tricks to determine the endianness of the machine. Unfortunately the compiler doesn't seem to be able to optimise the ordering operation, but reading in the weights is not a time-critical operation and the extra time it takes should not be noticeable.
Increases the use of NNUE evaluation in positions without captures/pawn moves,
by increasing the NNUEThreshold threshold with rule50_count.
This patch will force Stockfish to use NNUE eval more and more in materially
unbalanced positions, when it seems that the classical eval is struggling to
win and only manages to shuffle. This will ask the (slower) NNUE eval to
double-check the potential fortress branches of the search tree, but only
when necessary.
Do not show the details of the default architecture for a simple "make help"
invocation, as the details are most likely to confuse beginners. Instead we
make it clear which architecture is the default and put an example at the end
of the Makefile as an incentative to use "make help ARCH=blah" to discover
the flags used by the different architectures.
```
make help
make help ARCH=x86-64-ssse3
```
Also clean-up and modernize a bit the Makefile examples while at it.
Daylen Yang [Tue, 11 Aug 2020 19:02:48 +0000 (12:02 -0700)]
Use posix_memalign for Apple Silicon instead of _mm_malloc
fails to build on that target, because of missing Intel Intrinsics.
macOS has posix_memalign() since ~2014 so we can simplify the code and just use that for all Apple platforms.
SFisGOD [Mon, 10 Aug 2020 13:13:56 +0000 (21:13 +0800)]
Tweak castling extension
Change condition from three friendly pieces to two. This now means that we only extend castling on the king side if there are no other friendly pieces aside from king and rook. For the queen side, we only extend if there is only a rook and another friendly piece or if there is only a single rook and no other friendly piece but this is very rare.
Guy Vreuls [Fri, 7 Aug 2020 15:07:46 +0000 (17:07 +0200)]
This commit enables a mixed bench, to improve CI and allow for PGO (profile-build) of the NNUE part of the code.
Joint work gvreuls / vondele
* Download the default NNUE net in AppVeyor
* Download net in travis CI `make net`
* Adjust tests to cover more archs, speedup instrumented testing
* Introduce 'mixed' bench as default, with further options:
classical, NNUE, mixed.
mixed (default) and NNUE require the default net to be present,
which can be obtained with
```
make net
```
Further examples (first is equivalent to `./stockfish bench`):
Provide vectorized NNUE code for SSE2 and MMX targets
This patch allows old x86 CPUs, from AMD K8 (which the x86-64 baseline
targets) all the way down to the Pentium MMX, to benefit from NNUE with
comparable performance hit versus hand-written eval as on more modern
processors.
NPS of the bench with NNUE enabled on a Pentium III 1.13 GHz (using the
MMX code):
master: 38951
this patch: 80586
NPS of the bench with NNUE enabled using baseline x86-64 arch, which is
how linux distros are likely to package stockfish, on a modern CPU
(using the SSE2 code):
master: 882584
this patch: 1203945
mstembera [Mon, 10 Aug 2020 06:50:59 +0000 (23:50 -0700)]
Workaround using unaligned loads for gcc < 9
despite usage of alignas, the generated (avx2/avx512) code with older compilers needs to use
unaligned loads with older gcc (e.g. confirmed crash with gcc 7.3/mingw on abrok).
the stateInfo at the rootPos is no longer read-only, as the NNUE accumulator is part of it.
Threads can thus not share this object and need their own copy.
Vizvezdenec [Sun, 9 Aug 2020 18:39:46 +0000 (21:39 +0300)]
Adjust NNUE usage based on number of pawns in position
The idea of this patch is that positions are usually more complex and hard to evaluate even if there are more pawns.
This patch adjusts NNUE threshold usage depending on number of pawns in position, if pawn count is <3 we use the
classical evaluation more often, for pawn count = 3 patch the is non-functional,
with pawn count > 3 NNUE evaluation is used more often.
Guy Vreuls [Sat, 8 Aug 2020 10:45:10 +0000 (12:45 +0200)]
Parallelize Link Time Optimization for GCC, CLANG and MINGW
This patch tries to run multiple LTO threads in parallel, speeding up
the build process of optimized builds if the -j make parameter is used.
This mitigates the longer linking times of optimized builds since the
integration of the NNUE code. Roughly 2x build speedup.
I've tried a similar patch some two years ago but it ran into trouble
with old compiler versions then. Since we're on the C++17 standard now
these old compilers should be obsolete.
Vizvezdenec [Sat, 8 Aug 2020 14:43:41 +0000 (17:43 +0300)]
Do more aggressive futility pruning for captures
This patch lines up with other patches which use better eval to produce more aggressive cutoffs based on static evaluation of position, it allows more aggressive futility pruning for captures - so now we will be producing them with bigger evaluation of position, so more often.
Vizvezdenec [Sat, 8 Aug 2020 02:51:26 +0000 (05:51 +0300)]
Do more futility pruning for parent nodes.
This patch increases LMRdepth threshold for futility pruning at parent nodes so it can apply more often.
With radical change to evaluation approach it seems that search is really far from optimal state, especially it parts that use static evaluation of position.
SFisGOD [Fri, 7 Aug 2020 19:45:08 +0000 (03:45 +0800)]
Reintroduce late irreversible move extension
Reintroduce vondele's late irreversible move extension for fortress keeping.
This was removed when we only had classical eval.
Now that we have the NNUE net, it seems that this is useful again.
Lolligerhans [Fri, 7 Aug 2020 09:24:37 +0000 (11:24 +0200)]
Expand outposts to minors shielded by pawns
Allow any pawn in front of a minor piece to replace the pawn protection
requirement for outposts.
+-------+ +-------+
| . . o | | o . . | o Their pawns
| . o x | | o . . | x Our pawns
| o N . | | x o B | N,B New (reachable) outpost
| . . . | | . _ . | _ Reachable square behind a pawn
+-------+ +-------+
N outpost B reaches
outpost
We want outposts to be secured by pawns against major pieces. If
a minor is shielded by any pawn from above, it is rarely at the same
time protected by our pawn attacks from below. However, the pawn shield
in itself offers some degree of protection.
A pawn shield will now suffice to replace the pawn protection for the
outpost (and reachable outpost) bonus.
This effect stacks with the existing "minor behind pawn" bonus.
This alllows to simplify the code because the move counter haven't to be
decremented later if a move isn't legal. As a side effect now illegal
pruned moves doesn't included anymore in move counter. So slightly less
pruning and reductions are done.
MJZ1977 [Thu, 6 Aug 2020 15:39:10 +0000 (17:39 +0200)]
NNUE evaluation threshold
The idea is to use NNUE only on quite balanced material positions. This bring a big speedup on research since NNUE eval is slower than classical eval for most of the hardwares and specially on unbalanced positions with LazyEval.
nodchip [Wed, 5 Aug 2020 15:11:15 +0000 (17:11 +0200)]
Add NNUE evaluation
This patch ports the efficiently updatable neural network (NNUE) evaluation to Stockfish.
Both the NNUE and the classical evaluations are available, and can be used to
assign a value to a position that is later used in alpha-beta (PVS) search to find the
best move. The classical evaluation computes this value as a function of various chess
concepts, handcrafted by experts, tested and tuned using fishtest. The NNUE evaluation
computes this value with a neural network based on basic inputs. The network is optimized
and trained on the evalutions of millions of positions at moderate search depth.
The NNUE evaluation was first introduced in shogi, and ported to Stockfish afterward.
It can be evaluated efficiently on CPUs, and exploits the fact that only parts
of the neural network need to be updated after a typical chess move.
[The nodchip repository](https://github.com/nodchip/Stockfish) provides additional
tools to train and develop the NNUE networks.
This patch is the result of contributions of various authors, from various communities,
including: nodchip, ynasu87, yaneurao (initial port and NNUE authors), domschl, FireFather,
rqs, xXH4CKST3RXx, tttak, zz4032, joergoster, mstembera, nguyenpham, erbsenzaehler,
dorzechowski, and vondele.
This new evaluation needed various changes to fishtest and the corresponding infrastructure,
for which tomtor, ppigazzini, noobpwnftw, daylen, and vondele are gratefully acknowledged.
The first networks have been provided by gekkehenker and sergiovieri, with the latter
net (nn-97f742aaefcd.nnue) being the current default.
The evaluation function can be selected at run time with the `Use NNUE` (true/false) UCI option,
provided the `EvalFile` option points the the network file (depending on the GUI, with full path).
The performance of the NNUE evaluation relative to the classical evaluation depends somewhat on
the hardware, and is expected to improve quickly, but is currently on > 80 Elo on fishtest:
The needed networks can be found at https://tests.stockfishchess.org/nns
It is recommended to use the default one as indicated by the `EvalFile` UCI option.
Guidelines for testing new nets can be found at
https://github.com/glinscott/fishtest/wiki/Creating-my-first-test#nnue-net-tests
Integration has been discussed in various issues:
https://github.com/official-stockfish/Stockfish/issues/2823
https://github.com/official-stockfish/Stockfish/issues/2728
The integration branch will be closed after the merge:
https://github.com/official-stockfish/Stockfish/pull/2825
https://github.com/official-stockfish/Stockfish/tree/nnue-player-wip
We double the bonus for potential threats by minors and rooks against
our queen, in case of "queen vs pieces imbalance". Hopefully this will
improve a little bit the evaluation for this well-known Stockfish weakness.
This is a functional simplification that removes the conditional_more_than_two()
function, which was quite strange and kooky. Note the very minor change to the bench
value.
See this thread for relevant comments on the passing branch:
protonspring/Stockfish@d89730d...ff35b50
This corrects a functional change in
https://github.com/official-stockfish/Stockfish/commit/ddcbacd04d1c860e808202ce8c1206c8acdca627
changing evaluation of KPPvK. Bench remains unchanged at low depth
With this patch, 8/8/5k1p/8/7p/7K/8/8 b - - 1 11 is again correctly evaluated as a draw.
In some French games, Stockfish likes to bring the Knight to a bad outpost spot. This is evident in TCEC S18 Superfinal Game 63, where there is a Knight outpost on the queenside but is actually useless. Stockfish is effectively playing a piece down while holding ground against Leela's break on the kingside.
This patch turns the +56 mg bonus for a Knight outpost into a -7 mg penalty if it satisfies the following conditions:
* The outpost square is not on the CenterFiles (i.e. not on files C,D,E and F)
* The knight is not attacking non pawn enemies.
* The side where the outpost is located contains only few enemies, with a particular conditional_more_than_two() implementation
Thank you to apospa...@gmail.com for bringing this to our attention and for providing insights.
See https://groups.google.com/forum/?fromgroups=#!topic/fishcooking/dEXNzSIBgZU
Reference game: https://tcec-chess.com/#div=sf&game=63&season=18
various related tests show strong test results, but so far no generalizations or simplifications of conditional_more_than_two() are found. See PR for details.
Probcut is a heuristic that wasn't changed a lot in past years,
all attempts to change it using information / writing info to transposition table failed.
This patch has a number of differences that can be summarized as follows:
* For TT write/read we use depth - 3. Because probcut search is depth - 4 but we actually do the move prior to it so effectively we do depth - 3 search;
* In any case of depth of eval from transposition table being >= depth - 3 we either produce cutoff or refuse to even do probcut search, this is allowing us to write info of probcut to transposition table because we know that we wouldn't be overwriting some deeper data with our depth - 3 search - this is an important aspect of this patch;
* For some not really known reason this patch completely ignores tte->bound() - which was the case for previous patch that made probcut interact with TT, maybe 2) is the reason, although it's unproven.
However, an additional bugfix was needed to avoid checking a condition on ttMove if was not available. This passed non-regression bounds on top of the first version:
the goal of this PR is to better document how we process
the ep square (if any) given position fen command, and to
output more meaningful (and consistent) debug fen on the "d"
command. The implementation follows
https://en.wikipedia.org/wiki/X-FEN#Encoding_en-passant
following x-fen, it is "valid" to record ep even if ep would put king en prise.
Fix for overevaluated blocked pawns on the 5th and 6th rank.
This is a rewrite of the original idea that uses only two parameters.
Thanks to rocky640 for pointing this out.
UCI_ShowWDL might not be shown by GUIs that don't know the option,
but crash on the WDL output, effectively making it hard for users to
turn it off and run the engine. This sets it by default to false.
A number of engines, GUIs and tournaments start to report WDL estimates
along or instead of scores. This patch enables reporting of those stats
in a more or less standard way (http://www.talkchess.com/forum3/viewtopic.php?t=72140)
The model this reporting uses is based on data derived from a few million fishtest LTC games,
given a score and a game ply, a win rate is provided that matches rather closely,
especially in the intermediate range [0.05, 0.95] that data. Some data is shown at
https://github.com/glinscott/fishtest/wiki/UsefulData#win-loss-draw-statistics-of-ltc-games-on-fishtest
Making the conversion game ply dependent is important for a good fit, and is in line
with experience that a +1 score in the early midgame is more likely a win than in the late endgame.
Even when enabled, the printing of the info causes no significant overhead.
We lower the endgame value of the evaluation when we detect that there
is only one queen left on the board (more precisely, we use a scale
factor of 37/64, or about 0.58, for the endgame part of the evaluation).
Hopefully this helps a little bit for the assessment of positions with
queen imbalance, which are one of the well-known Stockfish weaknesses.
This patch increases the endgame value of pawns on the fifth rank.
The increase is very small (+1 evaluation point, about 0.005 pawn)
for the pawns on external columns (a-b-c-f-g-h) and a bit bigger
(+7 evaluation points, about 0.033 pawn) for the pawns on d5/e5.
mstembera [Fri, 26 Jun 2020 05:08:17 +0000 (22:08 -0700)]
Remove old zobrist trick for castling rights
Removes an 8 year old micro optimization aimed at 32-bit architectures
because back then doing an xor of a Key could not be done in one instruction.
See original commit here 821e1c7
mstembera [Sat, 27 Jun 2020 00:26:46 +0000 (17:26 -0700)]
Fix fragile code to use proper random 64 bit keys.
This fixes an old issue where we want to make a position unique but only
change a small number of bits in the key instead of all 64 of them randomly.
This is fragile and can lead to non uniqueness issues in the TT.
Key make_key(uint64_t seed) takes any integer and produces a unique random 64 bit key.
It is computationally efficient and is based on a congruential pseudo random number
generator using well tested constants by Donald Knuth
(see https://en.wikipedia.org/wiki/Linear_congruential_generator)
the option was, since at least 2014, not correctly implemented,
ignoring all dynamic adjustments to optimum time in search.
Instead of fixing it, remove it, no need to expose an option that
will influence time management negatively.
Vizvezdenec [Tue, 23 Jun 2020 11:55:52 +0000 (14:55 +0300)]
Do less futility pruning for captures.
The idea of this patch is that if capture can be described as
"less valuable piece takes more valuable piece" it's not really correct
to add only piece value of captured piece to static evaluation
since there can be more threats in other places and opponent can't really
do much but recapture our capturing piece which leaves us space for
more captures thus winning more material and increasing static eval.
FauziAkram [Mon, 22 Jun 2020 09:52:31 +0000 (12:52 +0300)]
Introduce bonus for queen infiltration
Idea is that queen feels much better when it can't be kicked away now
or later by pawn moves, especially in endgame. Special thanks to
Linmiao Xu for the original idea of this patch.
Vizvezdenec [Wed, 17 Jun 2020 02:36:30 +0000 (05:36 +0300)]
More reduction for evading pawn moves.
pawn moves are irreversable unlike other evading moves; pawn is the least valuable piece in the game.
So it makes a lot of sence to assume that evading pawn moves are on average not as good as other evading moves thus can be reduced more.
We increase a little bit the midgame value of pawns on a4, h4, a6 and h6.
Original idea by Malcolm Campbell, who tried the version restricted to the
pawns on the H column a couple of weeks ago and got a patch which almost
passed LTC. The current pull request just adds the same idea for pawns on
the A column.
Possible follow-ups: maybe tweak the a5/h5 pawn values, and/or add a malus
for very low king mobility in midgame?
Scheme back to being derived from https://lemire.me/blog/2016/06/27/a-fast-alternative-to-the-modulo-reduction/
Also the default optimized version of the index calculation now uses fewer instructions.
https://godbolt.org/z/Tktxbv
Might benefit from mulx (requires -mbmi2)
xoto10 [Sun, 14 Jun 2020 20:50:27 +0000 (21:50 +0100)]
Retuned values after eval quantize patch.
The last search tune patch was tested before the implementation of #2733 which
presumably changed the search characteristics noticeably. Another tuning run was
done, see https://tests.stockfishchess.org/tests/view/5ee5b434ca6c451633a9a08c
and the updated values passed these tests:
projects / stockfish / log