Update NNUE architecture to SFNNv6 with larger L1 size of 1536

Created by training a new net from scratch with L1 size increased from 1024 to 1536.
Thanks to Vizvezdenec for the idea of exploring larger net sizes after recent
training data improvements.

A new net was first trained with lambda 1.0 and constant LR 8.75e-4. Then a strong net
from a later epoch in the training run was chosen for retraining with start-lambda 1.0
and initial LR 4.375e-4 decaying with gamma 0.995. Retraining was performed a total of
3 times, for this 4-step process:

1. 400 epochs, lambda 1.0 on filtered T77+T79 v6 deduplicated data
2. 800 epochs, end-lambda 0.75 on T60T70wIsRightFarseerT60T74T75T76.binpack
3. 800 epochs, end-lambda 0.75 and early-fen-skipping 28 on the master dataset
4. 800 epochs, end-lambda 0.7 and early-fen-skipping 28 on the master dataset

In the training sequence that reached the new nn-8d69132723e2.nnue net,
the epochs used for the 3x retraining runs were:

1. epoch 379 trained on T77T79-filter-v6-dd.min.binpack
2. epoch 679 trained on T60T70wIsRightFarseerT60T74T75T76.binpack
3. epoch 799 trained on the master dataset

For training from scratch:

python3 easy_train.py \
  --experiment-name new-L1-1536-T77T79-filter-v6dd \
  --training-dataset /data/T77T79-filter-v6-dd.min.binpack \
  --max_epoch 400 \
  --lambda 1.0 \
  --start-from-engine-test-net False \
  --engine-test-branch linrock/Stockfish/L1-1536 \
  --nnue-pytorch-branch linrock/Stockfish/misc-fixes-L1-1536 \
  --tui False \
  --gpus "0," \
  --seed $RANDOM

Retraining commands were similar to each other. For the 3rd retraining run:

python3 easy_train.py \
  --experiment-name L1-1536-T77T79-v6dd-Re1-LeelaFarseer-Re2-masterDataset-Re3-sameData \
  --training-dataset /data/leela96-dfrc99-v2-T60novdecT80juntonovjanfebT79aprmayT78jantosepT77dec-v6dd.binpack \
  --early-fen-skipping 28 \
  --max_epoch 800 \
  --start-lambda 1.0 \
  --end-lambda 0.7 \
  --lr 4.375e-4 \
  --gamma 0.995 \
  --start-from-engine-test-net False \
  --start-from-model /data/L1-1536-T77T79-v6dd-Re1-LeelaFarseer-Re2-masterDataset-nn-epoch799.nnue \
  --engine-test-branch linrock/Stockfish/L1-1536 \
  --nnue-pytorch-branch linrock/nnue-pytorch/misc-fixes-L1-1536 \
  --tui False \
  --gpus "0," \
  --seed $RANDOM

The T77+T79 data used is a subset of the master dataset available at:
https://robotmoon.com/nnue-training-data/

T60T70wIsRightFarseerT60T74T75T76.binpack is available at:
https://drive.google.com/drive/folders/1S9-ZiQa_3ApmjBtl2e8SyHxj4zG4V8gG

Local elo at 25k nodes per move vs. nn-e1fb1ade4432.nnue (L1 size 1024):
nn-epoch759.nnue : 26.9 +/- 1.6

Failed STC
https://tests.stockfishchess.org/tests/view/64742485d29264e4cfa75f97
LLR: -2.94 (-2.94,2.94) <0.00,2.00>
Total: 13728 W: 3588 L: 3829 D: 6311
Ptnml(0-2): 71, 1661, 3610, 1482, 40

Failing LTC
https://tests.stockfishchess.org/tests/view/64752d7c4a36543c4c9f3618
LLR: -1.91 (-2.94,2.94) <0.50,2.50>
Total: 35424 W: 9522 L: 9603 D: 16299
Ptnml(0-2): 24, 3579, 10585, 3502, 22

Passed VLTC 180+1.8
https://tests.stockfishchess.org/tests/view/64752df04a36543c4c9f3638
LLR: 2.95 (-2.94,2.94) <0.50,2.50>
Total: 47616 W: 13174 L: 12863 D: 21579
Ptnml(0-2): 13, 4261, 14952, 4566, 16

Passed VLTC SMP 60+0.6 th 8
https://tests.stockfishchess.org/tests/view/647446ced29264e4cfa761e5
LLR: 2.94 (-2.94,2.94) <0.50,2.50>
Total: 19942 W: 5694 L: 5451 D: 8797
Ptnml(0-2): 6, 1504, 6707, 1749, 5

closes https://github.com/official-stockfish/Stockfish/pull/4593

bench 2222567

diff --git a/src/evaluate.h b/src/evaluate.h
index f5db1c1e6220441bc54cf41182e8f78095b4b0d9..0990111cf6d20f4056c612a880e5b67e9405fcc0 100644 (file)
--- a/src/evaluate.h
+++ b/src/evaluate.h
@@ -39,7 +39,7 @@ namespace Eval {
   // The default net name MUST follow the format nn-[SHA256 first 12 digits].nnue
   // for the build process (profile-build and fishtest) to work. Do not change the
   // name of the macro, as it is used in the Makefile.
-  #define EvalFileDefaultName   "nn-e1fb1ade4432.nnue"
+  #define EvalFileDefaultName   "nn-8d69132723e2.nnue"
 
   namespace NNUE {
 

diff --git a/src/nnue/nnue_architecture.h b/src/nnue/nnue_architecture.h
index 508f3aae0a7ab18a3551693cff18aedcb682e9e0..d10434f34b8df0d54312950c2fa1b241074840e2 100644 (file)
--- a/src/nnue/nnue_architecture.h
+++ b/src/nnue/nnue_architecture.h
@@ -39,7 +39,7 @@ namespace Stockfish::Eval::NNUE {
 using FeatureSet = Features::HalfKAv2_hm;
 
 // Number of input feature dimensions after conversion
-constexpr IndexType TransformedFeatureDimensions = 1024;
+constexpr IndexType TransformedFeatureDimensions = 1536;
 constexpr IndexType PSQTBuckets = 8;
 constexpr IndexType LayerStacks = 8;