31m059 [Sat, 23 Nov 2019 09:03:51 +0000 (04:03 -0500)]
King danger: retire attacked-by-bishop defense
In a recent commit, "Introduce king flank defenders," a term was introduced
by Michael Chaly (@Vizvezdenec) to reduce king danger based on king defenders,
i.e., friendly attacks on our King Flank and Camp. This is a powerful idea
and broadly applicable to all of our pieces.
An earlier, but narrower, version of a similar idea was already coded into
king danger, with a term reducing king danger simply if we had a bishop and
king attacking the same square -- there is also a similar term for knights,
but roughly three times larger. I had attempted to tweak this term's coefficient
fairly recently, in a series of tests in early September which increased this
coefficient. All failed STC with significantly negative scores.
Now that the king flank defenders term has been introduced, it appears that
the bishop-defense term can be simplified away without compensation or
significant Elo loss.
Where do we go from here? This PR is a natural follow-up to "Introduce king
flank defenders," which proposed simplification with existing and overlapping
terms, such as this one. That PR also mentioned that the coefficient it
introduced appeared arbitrary, so perhaps this PR can facilitate a tweak to
increase king flank defenders' coefficient.
Additionally, this pull request is extremely similar to https://github.com/official-stockfish/Stockfish/pull/1821,
which was (coincidentally) merged a year ago, to the day (November 23, 2018).
That patch also simplified away a linear king danger tropism term, which was
soon after replaced with a quadratic term by @Vizvezdenec (which would not have
passed without the simplification). @Vizvezdenec, again by coincidence, has
recently been trying to implement a quadratic term, this time for defenders
rather than attackers. This history of this evaluation code suggests that
this simplification might be enough to help a patch for quadratic king-flank
defenders pass.
Interesting discussion on Github about this pull request:
https://github.com/official-stockfish/Stockfish/pull/2424
---
This pull request was opened less than one week before the holiday of
Thanksgiving here in the United States. In keeping with the holiday
tradition of expressing gratitude, I would like to thank our generous
CPU donors, talented forum contributors, innovative developers, speedy
fishtest approvers, and especially our hardworking server maintainers
(@ppigazzini and @tomtor). Thank you all for a year of great Stockfish
progress!
SFisGOD [Wed, 20 Nov 2019 19:31:23 +0000 (03:31 +0800)]
Simplify endgame factor for opposite colored bishops
Stockfish is continually improving. Patches that gain elo in the past may
no longer be needed as stockfish improved elsewhere. This patch removes
passed pawns count dependence in opposite colored bishops scale factor.
We used the mean of passed count pawns (~1.4) to compensate, and changed
the base value from 16 to 22.
Vizvezdenec [Wed, 20 Nov 2019 20:20:20 +0000 (23:20 +0300)]
Do lmr for more captures
Based on machinery introduced by vondele. Logic behind patch if relatively simple -
if we reduce less with high hit rate of transposition table somewhat logical is to
reduce more with low hit rate. For example enable all captures for LMR.
Threshold 0.375 is arbitrary and can be tweaked :)
Alain SAVARD [Sat, 16 Nov 2019 19:42:47 +0000 (14:42 -0500)]
Outpost Endgame values
Remove the recent rank based Outpost array by using a weighted average value
computed using a frequency analysis by rank from a large set of middle game
positions.
The higher eg values introduced by the new Outpost array (which were about
twice the previous masters) are thus preserved.
Vizvezdenec [Sun, 17 Nov 2019 18:47:17 +0000 (21:47 +0300)]
Simplify advanced pawn push pruning
This patch simplifies away all conditions related to advanced pawn pushes
in shallow depth pruning. Idea is based on fact that in master we have
advanced pawn pushes not being pruned what we are only in PV node and
when non-pawn material of opponent is > Bishop, so pretty rarely. With
this patch we will have all pruning heuristics working for this moves as
for every other move.
This patch measures how frequently search is exploring new configurations.
This is done be computing a running average of ttHit. The ttHitAverage rate
is somewhat low (e.g. 30% for startpos) in the normal case, while it can be
very high if no progress is made (e.g. 90% for the fortress I used for testing).
This information can be used to influence search. In this patch, by adjusting
reductions if the rate > 50%. A first version (using a low ttHitAverageResolution
and this 50% threshold) passed testing:
However, as discussed in pull request 2414, using a larger ttHitAverageResolution
gives a better approximation of the underlying distributions. This needs a slight
adjustment for the threshold as the new distributions are shifted a bit compared
to the older ones, and this threshold seemingly is sensitive (we used 0.53125 here).
https://github.com/official-stockfish/Stockfish/pull/2414
This final version also passed testing, and is used for the patch:
Vizvezdenec [Sat, 16 Nov 2019 11:53:11 +0000 (14:53 +0300)]
Introduce king flank defenders
This patch implements what we have been trying for quite some time -
dependance of kingdanger on balance of attackers and defenders of king
flank, to avoid overestimate attacking power if the opponent has enough
defenders of king position. We already have some form of it in bishop
and knight defenders - this is further work in this direction.
What to do based on this?
1) constant 4 is arbitrary, maybe it is not optimal
2) maybe we can use quadratic formula as in kingflankattack
3) simplification into alrealy existing terms is always a possibility :)
4) overall kingdanger tuning always can be done.
SFisGOD [Tue, 12 Nov 2019 01:22:09 +0000 (09:22 +0800)]
Rank-based outposts
Introduce OutpostRank[RANK_NB] which contains a bonus according to
the rank of the outpost. We use it for the primary Outpost bonus.
The values are based on the trends of the SPSA tuning run with some
manual tweaks.
The inspiration for this patch came from Stefan Geschwentner's attempt
of modifying BishopPawns into a rank-based penalty. Michael Stembera
suggested that maybe the S(0, 0) ranks (3rd, 7th and also maybe 8th)
can still be tuned. This would expand our definition of Outpost and
OutpostRanks would be removed altogether. Special thanks to Mark Tenzer
for all the help and excellent suggestions.
Miguel Lahoz [Sun, 10 Nov 2019 09:49:06 +0000 (17:49 +0800)]
Shallow depth pruning on NonPV advanced pawn push
Usually advanced pawn pushes are not considered in shallow depth pruning
because it is risky to do so with possible promotions near the horizon.
However, this heuristic is not also beneficial on NonPV nodes since we
can afford to take slightly more risk on less important nodes.
The removed lines approximately duplicate equivalent logic in the movePicker.
Adjust the futility_move_count to componsate for some difference
(the movePicker prunes one iteration of the move loop later).
It would be interesting to see if we can extract some Elo gain
from the new futility_move_count formula, for instance by somehow
incorporating the final -1 in the 5 constant, or adding a linear
term to the quadratics...
Alain SAVARD [Sun, 27 Oct 2019 20:16:26 +0000 (16:16 -0400)]
Sequencing tweak in tbprobe()
Followup of "issue" #2372, which was in fact a small speed-up
proposal by user @d3vv for the probing code of tablebases.
See comments on this issue where it was proven by Alin Savard that the
proposed change is more efficient on average than master on all type of
sequences it will usually be called.
Note that on gcc 4.3, this will produce a bogus warning which was solved
with ulterior gcc versions: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=43949
```
setoption name Hash value 8
setoption name Threads value 1
position fen 8/1p2KP2/1p4q1/1Pp5/2P5/N1Pp1k2/3P4/1N6 b - - 76 40
go depth 49
```
even though the position is a draw. Generally, SF tries to display only
proven mate scores, so this is a bug.
This was posted http://www.talkchess.com/forum3/viewtopic.php?f=2&t=72166
by Uri Blass, with the correct analysis that this must be related to the
50 moves draw rule being ignored somewhere.
Indeed, this is possible as positions and there eval are stored in the TT,
without reference to the 50mr counter. Depending on the search path followed
a position can thus be mate or draw in the TT (GHI or Graph history interaction).
Therefore, to prove mate lines, the TT content has to be used with care. Rather
than ignoring TT content in general or for mate scores (which impact search or
mate finding), it is possible to be more selective. In particular, @WOnder93
suggested to only ignore the TT if the 50mr draw ply is closer than the mate
ply. This patch implements this idea, by clamping the eval in the TT to
+-VALUE_MATED_IN_MAX_PLY. This retains the TTmove, but causes a research of
these lines (with the current 50mr counter) as needed.
This patch hardly ever affects search (as indicated by the unchanged
bench), but fixes the testcase. As the conditions are very specific,
also mate finding will almost never be less efficient (testing welcome).
It was also shown to pass STC and LTC non-regression testing, in a form
using if/then/else instead of ternary operators:
This reverts the previous commit. The PSQT changes in this previous
commit originated from tests against quite an old version of master
which did not include the other PSQT changes of 474d133 for the other
pieces, and there might be some unknown interactions between the PSQT
tables. So we made a non-regression test of the last commit against the
last-but-one commit. This test failed, leading to the revert decision.
SFisGOD [Mon, 4 Nov 2019 19:06:41 +0000 (03:06 +0800)]
Rook PSQT Tuned
This patch uses about half the changes of the SPSA tuning run:
http://tests.stockfishchess.org/tests/view/5dba93d30ebc5925b64ed3bf
About a month ago, xoto10's patch raised the mg value of the third rank
center files from -1 to 7 to encourage rook lifts to the third rank. About
three days later, Rocky's patch lowered this value from 7 to 3. This patch
raises that again from 3 to 12 and ends up greater than the original rook
lift patch.
MichaelB7 [Sun, 3 Nov 2019 00:04:05 +0000 (20:04 -0400)]
Remove shuffle extension
It was noted in an earlier patch that all of the positions below needed the
Shuffle Detection idea to be solved:
3r4/p3r1pk/PpBb1pRp/1KpPpP1P/2P1P1R1/8/8/8 b - - 32 86
8/8/8/1k6/2p5/p1K5/N2B2r1/8 b - - 59 109
1r4k1/1r1bq3/4p1p1/3pPpPp/pNpN1P1P/P1PnQ3/1PK5/1R3R2 b - - 13 82
5k2/3b4/5p2/p1p1pPp1/PpPpP1Pp/1P1P3P/8/3R1K2 w - - 20 1
But Stockfish has envolved a bit since the Shuffle Detection patch introduction,
and this patch proves Stockfish is able to solves these drawn positions without it,
even on single core without EGTB.
Example of search by Michael Byrne for the FEN position:
q1B5/1P1q4/8/8/8/6R1/8/1K1k4 w - - 0 1
This position is win for white and the only moves that wins is Rg1 - all other moves
either draw or lose. With single core and 1024M hash, it is solved without shuffle
detection in 38 seconds on my machine (with no EGTB). This was the position that was
locked in a loop in the initial shuffle detection patch!
31m059 [Fri, 1 Nov 2019 04:27:19 +0000 (00:27 -0400)]
Make Square and Bitboard operators commutative
As Stockfish developers, we aim to make our code as legible and as close
to simple English as possible. However, one of the more notable exceptions
to this rule concerns operations between Squares and Bitboards.
Prior to this pull request, AND, OR, and XOR were only defined when the
Bitboard was the first operand, and the Square the second. For example,
for a Bitboard b and Square s, "b & s" would be valid but "s & b" would not.
This conflicts with natural reasoning about logical operators, both
mathematically and intuitively, which says that logical operators should
commute.
More dangerously, however, both Square and Bitboard are defined as integers
"under the hood." As a result, code like "s & b" would still compile and give
reasonable bench values. This trap occasionally ensnares even experienced
Stockfish developers, but it is especially dangerous for new developers not
aware of this peculiarity. Because there is no compilation or runtime error,
and a reasonable bench, only a close review by approvers can spot this error
when a test has been submitted--and many times, these bugs have slipped past
review. This is by far the most common logical error on Fishtest, and has
wasted uncountable STC games over the years.
However, it can be fixed by adding three non-functional lines of code. In this
patch, we define the operators when the operands are provided in the opposite
order, i.e., we make AND, OR, and XOR commutative for Bitboards and Squares.
Because these are inline methods and implemented identically, the executable
does not change at all.
This patch has the small side-effect of requiring Squares to be explicitly
cast to integers before AND, OR, or XOR with integers. This is only performed
twice in Stockfish's source code, and again does not change the executable at
all (since Square is an enum defined as an integer anyway).
For demonstration purposes, this pull request also inverts the order of one AND
and one OR, to show that neither the bench nor the executable change. (This
change can be removed before merging, if preferred.)
I hope that this pull request significantly lowers the barrier-of-entry for new
developer to join the Stockfish project. I also hope that this change will improve
our efficiency in using our generous CPU donors' machines, since it will remove
one of the most common causes of buggy tests.
Following helpful review and comments by Michael Stembera (@mstembera), we add
a further clean-up by implementing OR for two Squares, to anticipate additional
traps developers may encounter and handle them cleanly.
protonspring [Thu, 31 Oct 2019 15:01:33 +0000 (09:01 -0600)]
Simplify kingRing
Simplify the king ring initialization and make it more regular, by just
moving the king square off the edges and using PseudoAttacks by king from
this new square.
There is a small functional difference from the previous master, as the
old master excludes the original ksq square while this patch always includes
the nine squares block (after moving the king from the edges). Additionally,
master does not adjust the kingRing down if we are on relative rank 8,
while this patch treats all of the edges the same.
protonspring [Thu, 31 Oct 2019 16:17:46 +0000 (17:17 +0100)]
Consolidate pawn_push and up
This is a non-functional simplification. Pawn_push and Up are redundant.
If we make up pawn_push, we can use it for all of the Up's and Down's.
In this version, I've also left the Up and Down constants so that there
is no worse readability.
Make dynamic contempt weight factor dependent on static contempt so that higher
static contempt implies less dynamic contempt and vice versa. For default contempt
24 this is a non-functional change. But tests with contempt 0 shows an elo gain.
Also today is my birthday so i have already give to myself a gift with this patch :-)!
Further proceedings:
in the past we checked for default contempt that it doesn't regress against
contempt 0. Now that the later is stronger and the former is the same strength
this should be rechecked. Perhaps the default contempt have to be lowered.
It would be interesting to get some idea of the impact of this patch outside
of the 0-24 contempt range.
This PR refactors update_quiet_stats, update_capture_stats and search to more clearly reflect what is actually done.
Effectively, all stat updates that need to be done after search is finished and a bestmove is found,
are collected in a new function ```final_stats_update()```. This shortens our main search routine, and simplifies ```update_quiet_stats```.
The latter function is now more easily reusable with fewer arguments, as the handling of ```quietsSearched``` is only needed in ```final_stats_update```.
```update_capture_stats```, which was only called once is now integrated in ```final_stats_update```, which allows for removing a branch and reusing some ```stat_bonus``` calls. The need for refactoring was also suggested by the fact that the comments of ```update_quiet_stats``` and ```update_capture_stats``` were incorrect (e.g. ```update_capture_stats``` was called, correctly, also when the bestmove was a quiet and not a capture).
- Cleanups by Alain
- Group king attacks and king defenses
- Signature of futility_move_count()
- Use is_discovery_check_on_king()
- Simplify backward definition
- Use static asserts in move generator
- Factor a statement in move generator
Current master employs a scheme to adjust reductions on singular
nodes that is somewhat controversial, see
https://github.com/official-stockfish/Stockfish/pull/2167
This patch removes this use of a search result outside of [a,b],
by observing that the main effect of this code is to adjust the
reduction by an average of ~2 (1.7) rather than 1.
Stockfish crashes immediately if users enter a wrong file name (or even an existing
folder name) for debug log file. It may be hard for users to find out since it prints
nothing. If they enter the string via a chess GUI, the chess GUI may remember and
auto-send to Stockfish next time, makes Stockfish crashes all the time. Bug report by
Nguyen Hong Pham in this issue: https://github.com/official-stockfish/Stockfish/issues/2365
This patch avoids the crash and instead prefers to exit gracefully with a error
message on std:cerr, like we do with the fenFile for instance.
With the current questions and issues around threading, I had a look at
https://github.com/official-stockfish/Stockfish/issues/2299.
It seems there was a problem with data races when requesting eval via UCI while
a search was already running. To fix this an extra thread uithread was created,
presumably to avoid an overlap with Threads.main() that was causing problems.
Making this eval request seems to be outside the scope of UCI, and @vondele also
reports that the data race is not even fixed reliably by this change. I suggest
we simplify the threading here by removing this uithread and adding a comment
signaling that user should not request eval when a search is already running.
The current bench is missing a position with high 50 moves rule counter,
making most 'shuffle' tests based on 50mr > N seem non-functional.
This patch adds one FEN with high 50mr counter to address this issue
(taken from a recent tcec game).
Four new FENs:
- position with high 50mr counter
- tactical position with many captures, checks, extensions, fails high/low
- two losses by Stockfish in the S16 bonus games against Houdini
See the pull request for nice comments by @Alayan-stk-2 about each position
in bench: https://github.com/official-stockfish/Stockfish/pull/2338
31m059 [Mon, 7 Oct 2019 18:47:43 +0000 (14:47 -0400)]
No reachable outpost bonus for bishops
Previously, we used various control statements and ternary operators to divide
Outpost into four bonuses, based on whether the outpost was for a knight or
bishop, and whether it was currently an Outpost or merely a potential ("reachable")
one in the future. Bishop outposts, however, have traditionally been worth far
less Elo in testing. An attempt to remove them altogether passed STC, but failed LTC.
Here we include a narrower simplification, removing the reachable Outpost bonus
for bishops. This bonus was always suspect, given that its current implementation
conflicts directly with BishopPawns. BishopPawns penalizes our bishops based on the
number of friendly pawns on the same color of square, but by definition, Outposts
must be pawn-protected! This PR helps to alleviate this conceptual contradiction
without loss of Elo and with slightly simpler code.
On a code level, this allows us to simplify a ternary operator into the previous
"if" block and distribute a multiplication into an existing constant Score. On a
conceptual level, we retire one of the four traditional Outpost bonuses.
Alayan [Mon, 7 Oct 2019 17:02:33 +0000 (19:02 +0200)]
Adjust aspiration window with eval
This patch changes the base aspiration window size depending on the absolute
value of the previous iteration score, increasing it away from zero. This
stems from the observation that the further away from zero, the more likely
the evaluation is to change significantly with more depth. Conversely, a
tighter aspiration window is more efficient when close to zero.
A beneficial side-effect is that analysis of won positions without a quick
mate is less prone to waste nodes in repeated fail-high that change the eval
by tiny steps.
Future work : the values used in this patch were only a reasonable guess.
Further testing should unveil more optimal values. However, the aspiration
window is rather tight with a minimum of 21 internal units, so discrete
integers put a practical limitation to such tweaking.
More exotic experiments around the aspiration window parameters could also
be tried, but efficient conditions to adjust the base aspiration window size
or allow it to not be centered on the current evaluation are not obvious.
The aspiration window increases after a fail-high or a fail-low is another
avenue to explore for potential enhancements.
Alain SAVARD [Sun, 6 Oct 2019 22:48:19 +0000 (00:48 +0200)]
Adjust pawn span
Run as a simplification
a) insures that pawn attacks are always included in the pawn span
(this "fixes" the case where some outpost or reachable outpost
bonus were awarded on squares controlled by enemy pawns).
b) compute the full span only if not "backward" or not "blocked".
By looking at "blocked" instead of "opposed", we get a nice simpli-
fication and the "new" outpost detection is almost identical, except
a few borderline cases on rank 4.
Ondrej Mosnacek [Sun, 6 Oct 2019 07:57:20 +0000 (09:57 +0200)]
Make priorCapture a bool
It is always used as a bool, so let's make it a bool straight away.
We can always redefine it as a Piece in a later patch if we want
to use the piece type or the piece color.
VoyagerOne [Sat, 5 Oct 2019 14:42:36 +0000 (10:42 -0400)]
Introduce separate counter-move tables for captures
Enhance counter-move history table by adding a capture/no-capture dimension,
depending wether the previous move was a quiet move or a capture. This doubles
the size of the table but provides more accurate move ordering.
Brian Sheppard [Sat, 28 Sep 2019 20:27:23 +0000 (16:27 -0400)]
Eliminate ONE_PLY
Simplification that eliminates ONE_PLY, based on a suggestion in the forum that
support for fractional plies has never been used, and @mcostalba's openness to
the idea of eliminating it. We lose a little bit of type safety by making Depth
an integer, but in return we simplify the code in search.cpp quite significantly.
No functional change
------------------------------------------
The argument favoring eliminating ONE_PLY:
* The term “ONE_PLY” comes up in a lot of forum posts (474 to date)
https://groups.google.com/forum/?fromgroups=#!searchin/fishcooking/ONE_PLY%7Csort:relevance
* There is occasionally a commit that breaks invariance of the code
with respect to ONE_PLY
https://groups.google.com/forum/?fromgroups=#!searchin/fishcooking/ONE_PLY%7Csort:date/fishcooking/ZIPdYj6k0fk/KdNGcPWeBgAJ
* To prevent such commits, there is a Travis CI hack that doubles ONE_PLY
and rechecks bench
* Sustaining ONE_PLY has, alas, not resulted in any improvements to the
engine, despite many individuals testing many experiments over 5 years.
The strongest argument in favor of preserving ONE_PLY comes from @locutus:
“If we use par example ONE_PLY=256 the parameter space is increases by the
factor 256. So it seems very unlikely that the optimal setting is in the
subspace of ONE_PLY=1.”
There is a strong theoretical impediment to fractional depth systems: the
transposition table uses depth to determine when a stored result is good
enough to supply an answer for a current search. If you have fractional
depths, then different pathways to the position can be at fractionally
different depths.
In the end, there are three separate times when a proposal to remove ONE_PLY
was defeated by the suggestion to “give it a few more months.” So… it seems
like time to remove this distraction from the community.
See the pull request here:
https://github.com/official-stockfish/Stockfish/pull/2289
mstembera [Sat, 24 Aug 2019 22:04:41 +0000 (15:04 -0700)]
Remove temporary shelter array
Remove temporary array of shelters and avoid iterating over it each time to find
if the shelter values after castling are better than the current value.
Work done on top of https://github.com/official-stockfish/Stockfish/pull/2277
Tweak the pawn attack span for backward pawns and the zone behind
opponent opposing pawns. This is important in positional play and
one of weaknesses of the engine in recent high level games.
In lazySMP it makes sense to prune a little more, as multiple threads
search wider. We thus increase the prefactor of the reductions slowly
as a function of the threads. The prefactor of the log(threads) term
is a parameter, this pull request uses 1/2 after testing.
Note:
A larger prefactor (1) passed similar tests at STC and LTC (8 threads),
while a very large one (2) passed STC quickly but failed LTC (8 threads).
For the single-threaded case there is no functional change.
Extend castling independently of singular extension
A curious feature of Stockfish's current extension code is its repeated
use of "else if." In most cases, this makes no functional difference,
because no more than one extension is applied; once one extension has
been applied, the remaining ones can be safely ignored.
However, if most singular extension search conditions are true, except
"value < singularBeta", no non-singular extensions (e.g., castling) can
be performed!
Three tests were submitted, for three of Stockfish's four non-singular
extensions. I excluded the shuffle extension, because historically there
have been concerns about the fragility of its conditions, and I did not
want to risk causing any serious search problems.
- Modifying the passed pawn extension appeared roughly neutral at STC. At
best, it appeared to be an improvement of less than 1 Elo.
- Modifying check extension performed very poorly at STC
- Modifying castling extension (this patch) produced a long "yellow" run
at STC (insufficient to pass, but positive score) and a strong LTC.
In simple terms, prior to this patch castling extension was occasionally
not applied during search--on castling moves. The effect of this patch is
to perform castling extension on more castling moves. It does so without
adding any code complexity, simply by replacing an "else if" with "if" and
reordering some existing code.
- It seems strange to me that check extension performed so poorly -- clearly
some of the singular extension conditions are also very important for check
extension. I am not an expert in search, and I do not have any intuition
about which of the eight conditions is/are the culprit. I will try a
succession of eight STC tests to identify the relevant conditions, then try
to replicate this PR for check extension.
- Recent tests interacting with the castle extension may deserve retesting.
I will shortly resubmit a few of my recent castling extension tweaks, rebased
on this PR/commit.
My deepest thanks to @noobpwnftw for the extraordinary CPU donation, and to
all our other fishtest volunteers, who made it possible for a speculative LTC
to pass in 70 minutes!
Remove the RookOnPawn logic (for rook on rank 5 and above aligning with pawns
on same row or file) which was overlapping with a few other parameters.
Inspired by @31m059 interesting result hinting that a direct attack on pawns
instead of PseudoAttacks might work.
http://tests.stockfishchess.org/tests/view/5d89a7c70ebc595091801b8d
After a few attempts by me and @31m059, and some long STC greens but red LTC,
as a proof of concept I first tried a local SPSA at VSTC trying to tune related
rook psqt rows, and mainly some rook related stuff in evaluate.cpp.
Result was STC green, but still red LTC,
Finally a 100M fishtest SPSA at LTC proved successful both at STC and LTC.
All this was possible with the awesome fishtest contributors.
At some point, I had 850 workers on the last test !
As part of the investigation of the hang caused by an incorrect implementation
of condition_variable in libwinpthread, it was realized that our custom Mutex
implementation is no longer needed. Prior to lazySMP this custom implementation
resulted in a 30% speedup, but now no speed difference can be measured as no
mutex is used on the hot path in lazySMP.
Restore development version (revert previous commit)
Revert the previous patch now that the binary for the super-final
of TCEC season 16 has been sent.
Maybe the feature of showing the name of compiler will be added to the
master branch in the future. But we may use a cleaner way to code it, see
some ideas using the Makefile approach at the end of pull request #2327 :
https://github.com/official-stockfish/Stockfish/pull/2327
This patch changes the weight for counting supports of pawns
from 17 to 21. Hopefully Stockfish will accept to play a bit
more of closed or semi-closed positions.
Thanks to @snicolet for mentioning that Komodo does this a lot and
Stockfish doesn't, which gave me the idea for this patch, and to
@noobpwnftw for providing cores to fishtest which allowed very quick
testing.
Future work: perhaps this can be refined somehow to encourage this
on other files, my attempts have failed.
As in that case, this test was yellow at STC and LTC, but green at VLTC,
indicative of the fact that the higher the drawrate, the more likely this
idea is beneficial.
It seems there is no other way to specify stack size on std::thread than linker
flags and the effective flags are named differently in many toolchains. On
toolchains where pthread is always available, this patch changes the stack
size change in our C++ code via pthread to ensure a minimum stack size of 8MB,
instead of relying on linker defaults which may be platform-specific.
Also raises default stack size on OSX to current Linux default (8MB) just to
be safe.
This patch decreases the endgame scale factor using the 50 moves counter.
Looking at some games with this patch, it seems to have two effects on
the playing style:
1) when no progress can be made in late endgames (for instance in fortresses
or opposite bishops endgames) the evaluation will be largely tamed down
towards a draw value.
2) more interestingly, there is also a small effect in the midgame play because
Stockfish will panic a little bit if there are more than four consecutive
shuffling moves with an advantage: the engine will try to move a pawn or to
exchange a piece to keep the advantage, so the follow-ups of the position
will be discovered earlier by the alpha-beta search.
This patch finally introduces something that was tried for years: midgame score
dependance on complexity of position. More precisely, if the position is very
simplified and the complexity measure calculated in the initiative() function
is inferior to -50 by an amount d, then we add this value d to the midgame score.
One example of play of this patch will be (again!) 4 vs 3 etc same flank endgames
where sides have a lot of non-pawn material: 4 vs 3 draw mostly remains the same
draw even if we add a lot of equal material to both sides.
STC run was stopped after 200k games (and not converging):
LLR: -1.75 (-2.94,2.94) [0.50,4.50]
Total: 200319 W: 44197 L: 43310 D: 112812
http://tests.stockfishchess.org/tests/view/5d7cfdb10ebc5902d386572c
This is the first and not really precise version, a lot of other stuff can be
tried on top of it (separate complexity for middlegame, some more terms, even
simple retuning of values).
The recently-added RookOnQueenFile evaluation term (36e4a86) provided a bonus
for placing our rook on the same file as an enemy queen.
Here, we relax a condition in this bonus, broadening its effect to any queen.
It is also strategically desirable to place the rook on the same file as a friendly
queen, so the restriction on the queen's color is removed.
Follow-up to previous commit. Update the documentation for the user when using `make`,
to show the preferred bmi2 compile in the advanced examples section.
Note: I made a mistake in the previous commit comment, the documentation is shown when
using `make` or `make help`, not `make --help`.
The only change done to the Makefile to get a somewhat faster binary as
discussed in #2291 is to add -msse4 to the compile options of the bmi2 build.
Since all processors supporting bmi2 also support sse4 this can be done easily.
It is a useful step to avoid sending around custom and poorly tested builds.
The speedup isn't enough to pass [0,4] but it is roughly 1.15Elo and a LOS of 90%:
LLR: -2.95 (-2.94,2.94) [0.00,4.00]
Total: 93009 W: 20519 L: 20316 D: 52174
Also rewrite the documentation for the user when using `make --help`, so that
the order of architectures for x86-64 has the more performant build one on top.
Scale down complexity for almost unwinnable endgames
This patch greatly scales down complexity of endgames when the
following conditions are all true together:
- pawns are all on one flank
- stronger side king is not outflanking weaker side
- no passed pawns are present
This should improve stockfish evaluation of obvious draws 4 vs 3, 3 vs 2
and 2 vs 1 pawns in rook/queen/knight/bishop single flank endgames where
strong side can not make progress.
a) This could be a powerful idea for refining some parts of the evaluation
function, a bit like when we try quadratics or other equations to emphasize
certain situations (xoto10).
b) Some other combinaison values for this bonus can be done further, or
overall retuning of weight and offset while keeping the formula simple.
This is a simplification that integrated WeakLever into doubled pawns.
Since we already check for !support for Doubled pawns, it is trivial
to check for weak lever by just checking more_than_one(lever).
We also introduce the Score * bool operation overload to remove some
casts in the code.
Maintain best move counter at the root and allow there only moves which has a counter
of zero for Late Move Reduction. For compensation only the first three moves are excluded
from Late Move Reduction per default instead the first four moves.
What we can further do:
- here we use a simple counting scheme but perhaps some aging to fade out early iterations
could be helpful
- use the best move counter also at inner nodes for LMR and/or pruning
protonspring [Sat, 24 Aug 2019 06:16:20 +0000 (08:16 +0200)]
Improve signature of evaluate_shelter()
Remove one parameter in function evaluate_shelter(), making all
comparisons for castled/uncastled shelter locally in do_king_safety().
Also introduce BlockedStorm penalty.
Passed non-regression test at STC:
LLR: 2.95 (-2.94,2.94) [-3.00,1.00]
Total: 65864 W: 14630 L: 14596 D: 36638
http://tests.stockfishchess.org/tests/view/5d5fc80c0ebc5939d09f0acc
protonspring [Mon, 12 Aug 2019 14:42:28 +0000 (08:42 -0600)]
Consolidate CastlingSide and CastlingRights
This is a non-functional simplification that removes CastlingSide and
implements the functionality in CastlingRights (thanks to Jörg Oster
for a comment on the first version of this patch).
protonspring [Wed, 21 Aug 2019 01:52:18 +0000 (19:52 -0600)]
Tuned Futility Equation
@Vizvezdenec array suggested that alternate values may be better than current
master (see pull request #2270 ). I tuned some linear equations to more closely
represent his values and it passed. These futility values seem quite sensitive,
so perhaps additional Elo improvements can be found here.
Jean Gauthier [Wed, 14 Aug 2019 12:44:21 +0000 (08:44 -0400)]
Slight speep up fetching the endgame table
Replace calls to count(key) + operator[key] with a single call to find(key).
Replace the std::map with std::unordered_map which provide O(1) access,
although the map has a really small number of objects.
Test with [0..4] failed yellow:
TC 10+0.1
SPRT elo0: 0.00 alpha: 0.05 elo1: 4.00 beta: 0.05
LLR -2.96 [-2.94,2.94] (rejected)
Elo 1.01 [-0.87,3.08] (95%)
LOS 85.3%
Games 71860 [w:22.3%, l:22.2%, d:55.5%]
http://tests.stockfishchess.org/tests/view/5d5432210ebc5925cf109d61
This is the result of a 200k tuning run at LTC:
http://tests.stockfishchess.org/tests/view/5d3576b70ebc5925cf0e9e1e
which passed quickly at LTC:
LLR: 2.95 (-2.94,2.94) [0.50,4.50]
Total: 12954 W: 2280 L: 2074 D: 8600
http://tests.stockfishchess.org/tests/view/5d3ff3f70ebc5925cf0f87a2
STC failed, but second LTC at [0,4] passed easily:
LLR: 2.96 (-2.94,2.94) [0.00,4.00]
Total: 8004 W: 1432 L: 1252 D: 5320
http://tests.stockfishchess.org/tests/view/5d407cff0ebc5925cf0f9119
Further work?
No doubt some of these changes produce most of the gain and some are neutral
or even bad, so further testing on individual/groups of parameters changed
here might show more gains. It does look like these tests might need to be
at LTC though, so maybe not too practical to do. See the thread in the pull
request for an interesting discussion:
https://github.com/official-stockfish/Stockfish/pull/2260
Bug fix: always choose shortest mate in multithread mode
In current master, with the voting scheme the best thread selection may
pick a non mate or not the shortest mate thread. This patch fixes this bug.
Formatting suggestion by Jörg Oster.
Related past pull requests:
https://github.com/official-stockfish/Stockfish/pull/1074
https://github.com/official-stockfish/Stockfish/pull/1215
Passed a [-4..0] verification test with 3 threads:
LLR: 2.95 (-2.94,2.94) [-4.00,0.00]
Total: 57158 W: 11374 L: 11424 D: 34360
http://tests.stockfishchess.org/tests/view/5d22deb30ebc5925cf0caefd
Just one sample output to demonstrate the effect of this patch.
5 Threads, 1 GB Hash
+---+---+---+---+---+---+---+---+
| r | | b | | | r | k | |
+---+---+---+---+---+---+---+---+
| | | | n | | p | b | |
+---+---+---+---+---+---+---+---+
| | | p | | p | | p | |
+---+---+---+---+---+---+---+---+
| p | | | | | | P | |
+---+---+---+---+---+---+---+---+
| P | p | | | B | | N | Q |
+---+---+---+---+---+---+---+---+
| | q | | | | | P | |
+---+---+---+---+---+---+---+---+
| | | R | | | P | | |
+---+---+---+---+---+---+---+---+
| | | | R | | | K | |
+---+---+---+---+---+---+---+---+
Additional thanks to @locutus2 , @miguel-l and @xoto10 for fruitful discussion.
There may be some more elo there since this tweak was the first one and numbers
are more or less arbitrary.
Non functional simplification when we find the passed pawns in pawn.cpp
and some code clean up. It also better follows the pattern "flag the pawn"
and "score the pawn".
-------------------------
The idea behind the third condition for candidate passed pawn is a little
bit difficult to visualize. Just for the record, the idea is the following:
Consider White e5 d4 against black e6. d4 can (in some endgames) push
to d5 and lever e6. Thanks to this sacrifice, or after d5xe6, we consider
e5 as "passed".
However:
- if White e5/d4 against black e6/c6: d4 cannot safely push to d5 since d5 is double attacked;
- if White e5/d4 against black e6/d5: d4 cannot safely push to d5 since it is occupied.
This is exactly what the following expression does:
Protonspring had a successful functional simplification that removes the
PassedFile array using a simple linear equation.
Merge the additive term S(5, 10) of protonspring passed file simplification
(pull request https://github.com/official-stockfish/Stockfish/pull/2250)
into the PassedRank array. This harmless change has a different bench because
the candidate passer evaluation will always get less compared to #2250,
as we apply bonus = bonus /2.
This exploits the recent fractional Skill Level, and is a result from some discussion in #2221 and the older #758.
Basically, if UCI_LimitStrength is set, it will internally convert UCI_Elo to a matching fractional Skill Level.
The Elo estimate is based on games at TC 60+0.6, Hash 64Mb, 8moves_v3.pgn, rated with Ordo, anchored to goldfish1.13 (CCRL 40/4 ~2000).
Note that this is mostly about internal consistency, the anchoring to CCRL is a bit weak, e.g. within this tournament,
goldfish and sungorus only have a 200Elo difference, their rating difference on CCRL is 300Elo.
I propose that we continue to expose 'Skill Level' as an UCI option, for backwards compatibility.
The result of a tournament under those conditions are given by the following table, where the player name reflects the UCI_Elo.
This is a functional simplification that removes the std::pow from reduction. The resulting reduction values are within 1% of master.
This is a simplification because i believe an fp addition and multiplication is much faster than a call to std::pow() which is historically slow and performance varies widely on different architectures.
Just blockSq, not forward file. Bench: 3377831 (#2240)
This is another functional simplification to Stockfish passed pawn evaluation.
Stockfish evaluates some pawns which are not yet passed as "candidate" passed pawns, which are given half the bonus of fully passed ones. Prior to this commit, Stockfish considered a passed pawn to be a "candidate" if (a) it would not be a passed pawn if moved one square forward (the blocking square), or (b) there were other pawns (of either color) in front of it on the file. This latter condition used a fairly complicated method, forward_file_bb; here, rather than inspect the entire forward file, we simply re-use the blocking square. As a result, some pawns previously considered "candidates", but which are able to push forward, no longer have their bonus halved.
Simplification tests passed quickly at both STC and LTC. The results from both tests imply that this simplification is, most likely, additionally a small Elo gain, with a LTC likelihood of superiority of 87 percent.
Michael Chaly [Sun, 14 Jul 2019 12:41:28 +0000 (15:41 +0300)]
tviigg. (#2238)
Current master code made sence when we had 2 types of bonuses for protected path to queen. But it was simplified so we have only one bonus now and code was never cleaned.
This non-functional simplification removes useless defendedsquares bitboard and removes one bitboard assignment (defendedSquares &= attackedBy[Us][ALL_PIECES] + defendedSquares & blockSq becomes just attackedBy[Us][ALL_PIECES] & blockSq also we never assign defendedSquares = squaresToQueen because we don't need it).
So should be small non-functional speedup.
Passed simplification SPRT.
http://tests.stockfishchess.org/tests/view/5d2966ef0ebc5925cf0d7659
LLR: 2.95 (-2.94,2.94) [-3.00,1.00]
Total: 23319 W: 5152 L: 5034 D: 13133
In Stockfish, both the middlegame and endgame bonus for a passed pawn are calculated as a product of two factors. The first is k, chosen based on the presence of defended and unsafe squares. The second is w, a quadratic function of the pawn's rank. Both are only applied if the pawn's relative rank is at least RANK_4.
It does not appear that the complexity of a quadratic function is necessary for w. Here, we replace it with a simpler linear one, which performs equally at both STC and LTC.
Update continuation history after LMR-triggered full depth research.
Directly after a LMR-triggered full depth research, we update the
continuation history for quiet moves (but with only half stat bonus).
We recently added a bonus for double pawn attacks on unsupported enemy pawns,
on June 27. However, it is possible that the unsupported pawn may become a passer
by simply pushing forward out of the double attack. By rewarding double attacks,
we may inadvertently reward the creation of enemy passers, by encouraging both of
our would-be stoppers to attack the enemy pawn even if there is no opposing
friendly pawn on the same file.
Here, we revise this term to exclude passed pawns. In order to simplify the code
with this change included, we non-functionally rewrite Attacked2Unsupported to
be a penalty for enemy attacks on friendly pawns, rather than a bonus for our
attacks on enemy pawns. This allows us to exclude passed pawns with a simple
& ~e->passedPawns[Us], while passedPawns[Them] is not yet defined in this part
of the code.
This dramatically reduces the proportion of positions in which Attacked2Unsupported
is applied, to about a third of the original. To compensate, maintaining the same
average effect across our bench positions, we nearly triple Attacked2Unsupported
from S(0, 20) to S(0, 56). Although this pawn formation is rare, it is worth more
than half a pawn in the endgame!
projects / stockfish / log