Don't attempt probcut if ttMove is not good enough.
This idea is loosely based on xoroshiro idea about raisedBeta and ttmoves.
If our ttmove have low enough ttvalue and is deep enough (deeper than our probcut depth) it makes little sense to try probcut moves, since the ttMove already more or less failed to produce one according to transposition table.
This patch refines the recently introduced interaction between
the space bonus and the number of blocked pawns in a position.
* pawns count as blocked also if their push square is attacked by 2 enemy pawns;
* overall dependence is stronger as well as offset;
* bonus increase is capped at 9 blocked pawns in position;
Scale up space weight with number of blocked pawns
This idea is loosely based on stockfish losses in closed positions in different tournaments. Space weight symmetrically increases for both sides the more blocked position is.
+-------+
| o . . | o their pawns
| x . . | x our pawns
| . x . | <- Can sacrifice to create passer?
+-------+
yes
1 2 3 4 5
+-------+ +-------+ +-------+ +-------+ +-------+
| o . . | | o r . | | o r . | | o . b | | o . b | lowercase: theirs
| x b . | | x . . | | x . R | | x . R | | x . . | uppercase: ours
| . x . | | . x . | | . x . | | . x . | | . x B |
+-------+ +-------+ +-------+ +-------+ +-------+
no no yes no yes
The value of our top pawn depends on our ability to advance our bottom
pawn, levering their blocker. Previously, this pawn configuration was
always scored as passer (although a blocked one).
Add requirements for the square s above our (possibly) sacrificed pawn:
- s must not be occupied by them (1).
- If they attack s (2), we must attack s (3).
- If they attack s with a minor (4), we must attack s with a minor (5).
The attack from their blocker is ignored because it is inherent in the
structure; we are ok with sacrificing our bottom pawn.
if these are ttMoves and played in positions with a high value of the rule50 counter. The unusual extension of 2 is safe in this context as awarding it will reset the rule50 counter, making sure it is awarded very rarely in a search path.
This patch partially addresses https://github.com/official-stockfish/Stockfish/issues/2620 as it should make it less likely to play a move that resets the counter, but that is worse than alternative moves after a slightly deeper search.
In master, if the received ttMove meets the prescribed conditions in the various MovePicker constructors, it is returned as the first move, otherwise we set it to MOVE_NONE. If set to MOVE_NONE, we no longer track what the ttMove was, and it will might be returned later in a list of generated moves. This may be a waste. With this patch, if the ttMove fails to meet the prescribed conditions, we simply skip the TT stages, but still store the move and make sure it's never returned.
This patch introduces a heuristic that is similar to countermove based pruning but for captures - capture history pruning. The idea is that we can (almost) safely prune really late captures with negative history if they don't give check so will most likely not produce some king-attacking tactic.
Before this commit, some pawns were considered "candidate" passed pawns and given half bonus. After this commit, all of these pawns are scored as passed pawns, and they do not receive less bonus.
This PR and commit are dedicated to our colleague Stefan Geschwentner (@locutus2), one of the most respected and accomplished members of the Stockfish developer community. Stockfish is a volunteer project and has always thrived because of Stefan's talent, insight, generosity, and dedication. Welcome back, Stefan!
protonspring [Tue, 31 Mar 2020 21:08:55 +0000 (15:08 -0600)]
remove KNPK endgame code
In more than 100k local KNPK games, there is no discernible difference between master and master with this endgame removed: master:42971, patch:42973, draws: 3969. Removal does not seem to regress in normal games.
The idea behind this patch is that if static eval is really bad so capturing of current piece on spot will still produce a position with an eval much lower than alpha then our best chance is to create some kind of king attack. So captures without check are mostly worse than captures with check and can be reduced more.
Vizvezdenec [Sun, 29 Mar 2020 17:04:20 +0000 (20:04 +0300)]
Count only the most advanced passed pawn for each file.
This patch adjusts definition of passed pawns - if there is a pawn of our color in the same file in front of a current pawn it's no longer counts as passed.
Vizvezdenec [Fri, 20 Mar 2020 09:12:56 +0000 (12:12 +0300)]
Adjust singular extension search depth
This patch applies a different singular extension search logic in case the position is ttPv && !PvNode.
It changes the depth of this search, higher for this types of nodes, and lower for other nodes.
Vizvezdenec [Tue, 17 Mar 2020 16:38:21 +0000 (19:38 +0300)]
Adjust singular LMR for positions seen in PV
This patch continues work on altering search for ttPv nodes, using recent idea to alter it more in not PvNodes. Previous tweak based on this idea adjusted singularBeta - this one adjusts value of singularLMR, so they are both related to singular extension search.
Reduce the "bad bishop" penalty when the bishop is protected by
one of our pawns, as it may indicate that the bishop has found
a safe spot outside the pawn chain.
protonspring [Mon, 9 Mar 2020 21:11:08 +0000 (22:11 +0100)]
Equations for edges and corners.
This is a functional simplification that removes the large arrays in endgames.cpp.
It also fixes a recently introduced bug (960d59d54143d84aab26deae65279a611fc989f4) in KNBvK,
now using flip_file() instead of ~.
One fen added to bench to increase endgame coverage.
Gary Heckman [Thu, 5 Mar 2020 17:37:08 +0000 (12:37 -0500)]
Fix ambiguity between clamp implementations
There is an ambiguity between global and std clamp implementations when compiling in c++17,
and on certain toolchains that are not strictly conforming to c++11.
This is solved by putting our clamp implementation in a namespace.
protonspring [Thu, 5 Mar 2020 19:07:48 +0000 (12:07 -0700)]
Cleanup KBPsK endgame
* Clarify distinction between strong side pawns and all pawns.
* Simplify and speed-up determination of pawns on the same file.
* Clarify comments.
* more_than_one() is probably faster than pos.count.
protonspring [Tue, 3 Mar 2020 00:32:02 +0000 (17:32 -0700)]
Use equations for PushAway and PushClose
A functional simplification replacing the corresponding arrays. Tested in two variants,
also the simpler one performs well, even though differences to master should be minimal.
AndyGrant [Mon, 24 Feb 2020 22:32:17 +0000 (23:32 +0100)]
Fix TT write in MultiPV case.
fixes an error reported earlier as https://github.com/official-stockfish/Stockfish/issues/2404 by @AndyGrant.
MultiPV at root shouldn't write to the TT for later lines, as that is neither the eval nor the bestmove for that position.
Fixing this error doesn't matter for playing games (http://tests.stockfishchess.org/tests/view/5dcdbd810ebc590256324a11).
However, it can lead to wrong mate announcements as reported by @uriblass. In particular the following testcase gives
wrong results for the second search, prior to this patch:
```
setoption name MultiPV value 2
position fen 5R2/2kB2p1/p2bR3/8/3p1B2/8/PPP5/2K5 b - - 0 49
go depth 40
position fen 2B2R2/3r2p1/p1kbR3/8/3p1B2/8/PPP5/2K5 b - - 8 48
go depth 40
```
31m059 [Sun, 23 Feb 2020 02:27:32 +0000 (21:27 -0500)]
Use single param for Outpost and ReachableOutpost.
In November 2019, as a result of the simplification of rank-based outposts by 37698b0,
separate bonuses were introduced for outposts that are currently occupied and outposts
that are reachable on the next move. However, the values of these two bonuses are
quite similar, and they have remained that way for three months of development.
It appears that we can safely retire the separate ReachableOutpost parameter and
use the same Outpost bonus in both cases, restoring the basic principles of Stockfish
outpost evaluation to their pre-November state, while also reducing
the size of the parameter space.
Günther Demetz [Fri, 21 Feb 2020 13:01:59 +0000 (14:01 +0100)]
Improve move order near the root
Current move histories are known to work well near the leaves, whilst at
higher depths they aren't very helpful. To address this problem this
patch introduces a table dedicated for what's happening at plies 0-3.
It's structured like mainHistory with ply index instead of color.
It get cleared with each new search and is filled during iterative
deepening at higher depths when recording successful quiet moves near
the root or traversing nodes which were in the principal variation
(ttPv).
Theory: KNNK is a dead draw, however the presence of the additional weakSide pawn opens up some mate opportunities. The idea is to block the pawn (preferably behind the Troitsky line) with one of the knights and press the weakSide king into a corner. If we can stalemate the king, we release the pawn with the knight (to avoid actual stalemate), and use the knight to complete the mate before the pawn promotes. This is also why there is an additional penalty for advancement of the pawn.
the code that prevents false mate announcements depending on the TT
state (GHI), incorrectly used VALUE_MATE_IN_MAX_PLY. The latter
constant, however, also includes, counterintuitively, the TB win range.
This patch fixes that, by restoring the behavior for TB win scores,
while retaining the false mate correctness, and improving the mate
finding ability. In particular
no alse mates are announced with the poisened hash testcase
```
position fen 8/8/8/3k4/8/8/6K1/7R w - - 0 1
go depth 40
position fen 8/8/8/3k4/8/8/6K1/7R w - - 76 1
go depth 20
ucinewgame
```
mates are found with the testcases reported in #2543
```
position fen 4k3/3pp3/8/8/8/8/2PPP3/4K3 w - - 0 1
setoption name Hash value 1024
go depth 55
ucinewgame
```
and
```
position fen 4k3/4p3/8/8/8/8/3PP3/4K3 w - - 0 1
setoption name Hash value 1024
go depth 45
ucinewgame
```
furthermore, on the mate finding benchmark (ChestUCI_23102018.epd),
performance improves over master, roughly reaching performance with the
false mate protection reverted
```
Analyzing 6566 mate positions for best and found mates:
On a testcase with TB enabled, progress is made consistently, contrary
to master
```
setoption name SyzygyPath value ../../../syzygy/3-4-5/
setoption name Hash value 2048
position fen 1R6/3k4/8/K2p4/4n3/2P5/8/8 w - - 0 1
go depth 58
ucinewgame
```
The incorrectly named VALUE_MATE_IN_MAX_PLY and VALUE_MATED_IN_MAX_PLY
were renamed into VALUE_TB_WIN_IN_MAX_PLY and VALUE_TB_LOSS_IN_MAX_PLY,
and correclty defined VALUE_MATE_IN_MAX_PLY and VALUE_MATED_IN_MAX_PLY
were introduced.
One further (corner case) mistake using these constants was fixed (go
mate X), which could lead to a premature return if X > MAX_PLY / 2,
but TB were present.
Thanks to @svivanov72 for one of the reports and help fixing the issue.
Vizvezdenec [Fri, 7 Feb 2020 17:04:43 +0000 (20:04 +0300)]
Modify singular beta for ttPv positions.
This patch lowers singular beta for positions that have been in pv and are not pv nodes.
The idea of using ttpv && !PvNode improved scaling with TC and could be
useful for other search heuristics.
can trigger an abort when compiling with debug=yes, and using 7men TB.
The assert should check that less than 8 pieces are in the key for each
side, matching the assumption that underlies the FEN string construction.
Also take explicitly care of a 'v' character in material strings.
Fixes an issue reported in the forum:
https://groups.google.com/d/msg/fishcooking/yoVC7etIpz0/7mS7ntZMBAAJ
joergoster [Mon, 27 Jan 2020 17:53:25 +0000 (18:53 +0100)]
Simplify hashfull calculation.
We can simplify the calculation of the hashfull info by looping over exact 1,000 entries,
and then divide the result by ClusterSize. Somewhat memory accesses, somewhat more accurate.
Guenther Demetz [Tue, 28 Jan 2020 12:38:03 +0000 (13:38 +0100)]
More bonus for bestMoves on past PV nodes
It looks like it is important to keep past PV (ttPv) nodes as close as possible to current PV nodes.
Credits to Mark Tenzer (31m059) & Stefan Geschwentner who first tried ideas on ttPv nodes.
31m059 [Tue, 28 Jan 2020 01:48:01 +0000 (20:48 -0500)]
Less NMP if the position was previously in PV.
The intention of the patch is to avoid aggressive null move pruning (NMP)
in positions that have previously been found to be important (PV nodes).
If we already do not apply NMP for current PV nodes, it makes sense to apply
it less often for positions that have previously been PV nodes too.
On my box, huge pages have a significant perf impact when using a big
hash size. They also speed up TT initialization big time:
vanilla (s) huge pages (s) speed-up
=======================================================================
time stockfish bench 16384 1 1 | 5.37 1.48 3.6x
In practice, huge pages with auto-defrag may always be enabled in the
system, in which case this patch has no effect. This
depends on the values in /sys/kernel/mm/transparent_hugepage/enabled
and /sys/kernel/mm/transparent_hugepage/defrag.
Use a SEE pruned capture move for history updates: this patch collects
pruned capture moves also in the failed captures list, so that they get
an update in capture history.
Vizvezdenec [Wed, 22 Jan 2020 00:54:44 +0000 (03:54 +0300)]
Tweak trapped rook penalty
This patch greatly increases the endgame penalty for having a trapped rook.
Idea was a result of witnessing Stockfish losing some games at CCCC exchanging
pieces in the position with a trapped rook which directly lead to a lost endgame.
This patch should partially fix such behavior making this penalty high even in
deep endgames.
Congrats! I think this might be a common pattern - scores that seem to mainly apply
to the midgame are often better with a similar (or at least fairly big) endgame value
as well. Maybe there are others eval parameters we can tweak like this...
Guenther Demetz [Thu, 16 Jan 2020 07:39:20 +0000 (08:39 +0100)]
Use (strict) greater-than-operator for 'improving'
Currently on a normal bench run in ~0,7% of cases 'improving' is set to
true although the static eval isn't improving at all, just keeping
equal. It looks like the strict gt-operator is more appropriate here,
since it returns to 'improving' its literal meaning without sideffects.
protonspring [Thu, 23 Jan 2020 17:18:58 +0000 (18:18 +0100)]
Use a std::bitset for KPKBitbase
This is a non-functional simplification. Looks like std::bitset works good
for the KPKBitbase. Thanks for Jorg Oster for helping get the speed up
(the [] accessor is faster than test()).
Speed testing: 10k calls to probe:
master 9.8 sec
patch 9.8 sec.
protonspring [Mon, 23 Dec 2019 17:58:30 +0000 (10:58 -0700)]
Determine opposite colors mathematically
This is a non-functional speed-up: master has to access SquareBB twice while this patch
determines opposite_colors just using the values of the squares. It doesn't seem to change
the overall speed of bench, but calling opposite_colors(...) 10 Million times:
master: 39.4 seconds
patch: 11.4 seconds.
The only data point I have (other than my own tests), is a quite old failed STC test:
LLR: -2.93 (-2.94,2.94) [-1.50,4.50]
Total: 24308 W: 5331 L: 5330 D: 13647
Ptnml(0-2): 315, 2577, 6326, 2623, 289
http://tests.stockfishchess.org/tests/view/5e010256c13ac2425c4a9a67
protonspring [Fri, 10 Jan 2020 22:08:47 +0000 (15:08 -0700)]
Simplify KPK classify
This is a non-functional simplification. If we use the "side to move" of the entry
instead of the template, one of the classify methods goes away. Furthermore, I've
resolved the colors in some of the statements (we're already assuming direction
using NORTH), and used stm (side to move) instead of "us," since this is much clearer
to me.
This is not tested because it is non-functional, only applies building the bitbase
and there are no changes to the binary (on my machine).
protonspring [Sat, 21 Dec 2019 22:36:29 +0000 (15:36 -0700)]
Simplify signature of remove_piece()
This is a non-functional simplification. Instead of passing the piece type
for remove_piece, we can rely on the board. The only exception is en-passant
which must be explicitly set because the destination square for the capture
is not the same as the piece to remove.
Verified also in the Chess960 castling case by running a couple of perft, see
the pull request discussion: https://github.com/official-stockfish/Stockfish/pull/2460
It is our pleasure to release Stockfish 11 to our fans and supporters.
Downloads are freely available at http://stockfishchess.org/download/
This version 11 of Stockfish is 50 Elo stronger than the last version, and
150 Elo stronger than the version which famously lost a match to AlphaZero
two years ago. This makes Stockfish the strongest chess engine running on
your smartphone or normal desktop PC, and we estimate that on a modern four
cores CPU, Stockfish 11 could give 1:1000 time odds to the human chess champion
having classical time control, and be on par with him. More specific data,
including nice cumulative curves for the progression of Stockfish strength
over the last seven years, can be found on [our progression page][1], at
[Stefan Pohl site][2] or at [NextChessMove][3].
In October 2019 Stockfish has regained its crown in the TCEC competition,
beating in the superfinal of season 16 an evolution of the neural-network
engine Leela that had won the previous season. This clash of style between an
alpha-beta and an neural-network engine produced spectacular chess as always,
with Stockfish [emerging victorious this time][0].
Compared to Stockfish 10, we have made hundreds of improvements to the
[codebase][4], from the evaluation function (improvements in king attacks,
middlegame/endgame transitions, and many more) to the search algorithm (some
innovative coordination methods for the searching threads, better pruning of
unsound tactical lines, etc), and fixed a couple of bugs en passant.
Our testing framework [Fishtest][5] has also seen its share of improvements
to continue propelling Stockfish forward. Along with a lot of small enhancements,
Fishtest has switched to new SPRT bounds to increase the chance of catching Elo
gainers, along with a new testing book and the use of pentanomial statistics to
be more resource-efficient.
Overall the Stockfish project is an example of open-source at its best, as
its buzzing community of programmers sharing ideas and daily reviewing their
colleagues' patches proves to be an ideal form to develop innovative ideas for
chess programming, while the mathematical accuracy of the testing framework
allows us an unparalleled level of quality control for each patch we put in
the engine. If you wish, you too can help our ongoing efforts to keep improving
it, just [get involved][6] :-)
Stockfish is also special in that every chess fan, even if not a programmer,
[can easily help][7] the team to improve the engine by connecting their PC to
Fishtest and let it play some games in the background to test new patches.
Individual contributions vary from 1 to 32 cores, but this year Bojun Guo
made it a little bit special by plugging a whole data center during the whole
year: it was a vertiginous experience to see Fishtest spikes with 17466 cores
connected playing [25600 games/minute][8]. Thanks Guo!
Vizvezdenec [Sun, 12 Jan 2020 23:59:06 +0000 (02:59 +0300)]
Tweak futility pruning constants
Based on recent improvement of futility pruning by @locutus2 : we lower
the futility margin to apply it for more nodes but as a compensation
we also lower the history threshold to apply it to less nodes. Further
work in tweaking constants can always be done - numbers are guessed
"by hand" and are not results of some tuning, maybe there is some more
Elo to squeeze from this part of code.
xoto10 [Sat, 11 Jan 2020 22:10:22 +0000 (22:10 +0000)]
Smarter time management near stop limit
This patch makes Stockfish search same depth again if > 60% of optimum time is
already used, instead of trying the next iteration. The idea is that the next
iteration will generally take about the same amount of time as has already been
used in total. When we are likely to begin the last iteration, as judged by total
time taken so far > 0.6 * optimum time, searching the last depth again instead of
increasing the depth still helps the other threads in lazy SMP and prepares better
move ordering for the next moves.
This updates estimates from 1.5 year ago, and adds missing terms. All estimates
from tests run on fishtest at 10+0.1 (STC), 20000 games, error bars +- 3 Elo,
see the original message in the pull request for the full list of tests.
Noteworthy changes are step 7 (futility pruning) going from ~30 to ~50 Elo
and step 13 (pruning at shallow depth) going from ~170 to ~200 Elo.
Full list of tests: https://github.com/official-stockfish/Stockfish/pull/2401
@Rocky640 made the suggestion to look at time control dependence of these terms.
I picked two large terms (early futility pruning and singular extension), so with
small relative error. It turns out it is actually quite interesting (see figure 1).
Contrary to my expectation, the Elo gain for early futility pruning is pretty time
control sensitive, while singular extension gain is not.
Figure 1: TC dependence of two search terms
Going back to the old measurement of futility pruning (30 Elo vs today 50 Elo),
the code is actually identical but the margins have changed. It seems like a nice
example of how connected terms in search really are, i.e. the value of early futility
pruning increased significantly due to changes elsewhere in search.
joergoster [Thu, 12 Dec 2019 11:53:47 +0000 (12:53 +0100)]
50-moves rule improvement for transposition table
User "adentong" reported recently of a game where Stockfish blundered a game
in a tournament because during a search there was an hash-table issue for
positions inside the tree very close to the 50-moves draw rule. This is part
of a problem which is commonly referred to as the Graph History Interaction (GHI),
and is difficult to solve in computer chess because storing the 50-moves counter
in the hash-table loses Elo in general.
Links:
Issue 2451 : https://github.com/official-stockfish/Stockfish/issues/2451
About the GHI : https://www.chessprogramming.org/Graph_History_Interaction
This patch tries to address the issue in this particular game and similar
reported games: it prevents that values from the transposition table are
getting used when the 50-move counter is close to reaching 100 (). The idea
is that in such cases values from previous searches, with a much lower 50-move
count, become less and less reliable.
More precisely, the heuristic we use in this patch is that we don't take the
transposition table cutoff when we have reached a 45-moves limit, but let the
search continue doing its job. There is a possible slowdown involved, but it will
also help to find either a draw when it thought to be losing, or a way to avoid
the draw by 50-move rule. This heuristics probably will not fix all possible cases,
but seems to be working reasonably well in practice while not losing too much Elo.
lantonov [Wed, 1 Jan 2020 08:10:39 +0000 (10:10 +0200)]
Tuned nullmove search
Tuning was done with Bayesian optimisation and sequential use of gaussian process
regressor and gaussian process classifier. The latter is used in lieu of ordinal
categorical modelling. Details will be given in Fishcooking forum topic: https://groups.google.com/forum/?fromgroups=#!topic/fishcooking/b3uhBBJcJG4
Vizvezdenec [Fri, 3 Jan 2020 02:53:59 +0000 (05:53 +0300)]
Introduce king infiltration bonus
Add king infiltration bonus to initiative calculation. Idea is somewhat similar
to outflanking - endgames are hard to win if each king is on it side of the board.
So this adds extra bonus for one of kings crossing the middle line.
Alain SAVARD [Sat, 4 Jan 2020 18:54:35 +0000 (13:54 -0500)]
Use a faster implementation of Static Exchange Evaluation
SEE (Static Exchange Evaluation) is a critical component, so we might
indulge some tricks to make it faster. Another pull request #2469 showed
some speedup by removing templates, this version uses Ronald de Man
(@syzygy1) SEE implementation which also unrolls the for loop by
suppressing the min_attacker() helper function and exits as soon as
the last swap is conclusive.
See Ronald de Man version there:
https://github.com/syzygy1/Cfish/blob/master/src/position.c
And since we are using new SPRT statistics, and that both pull requests
finished with less than 20000 games I also tested against master as
a speed-up:
lantonov [Mon, 9 Dec 2019 18:50:47 +0000 (20:50 +0200)]
Tuned razor and futility margins
Tuning was done with Bayesian optimisation with the following parameters:
Acquisition function: Expected Improvement
alpha: 0.05
xi: 1e-4
TC: 60+0.6
Number of iterations: 100
Initial points: 5
Batch size: 20 games
For the record, the idea was to run an experimental tuning with disabled
castling in the hope to get more hits on the TrappedRook and the king in
the c1- f1-f2-c2 area
http://tests.stockfishchess.org/tests/view/5dec57be51219d7befdc76e1
A first interpretation of that tuning was green STC (0, 4) and yellow LTC (0, 4):
http://tests.stockfishchess.org/tests/view/5ded04bc51219d7befdc773a
http://tests.stockfishchess.org/tests/view/5ded1e7a51219d7befdc7760
Thank you @xoto for trying this. Indeed, because the tuned Kc2 and Kf2 values
were quite different, it was a good idea to try something more neutral.
Vizvezdenec [Sun, 8 Dec 2019 14:10:14 +0000 (17:10 +0300)]
Exclude blockers for king from mobility area
This patch excludes blockers for king from mobility area. It was tried a couple
of times by now but now it passed. Performance is not enormously good but this
patch makes a lot of sence - blockers for king can't really move until king moves
(in most cases) so logic behind it is the same as behind excluding king square
from mobility area.
Vizvezdenec [Sat, 7 Dec 2019 14:56:33 +0000 (17:56 +0300)]
Do last capture extensions for every single node
This patch simplifies latest @MJZ1977 elo gainer. Seems like PvNode check in
condition of last capture extension is not needed. Note - even if this is a
simplification it actually causes this extension to be applied more often, thus
strengthening effect of @MJZ1977's patch.
projects / stockfish / log