avcodec/dstdec: Check that AC probabilities are within range

ISO/IEC 14496-3:2005(E): "Each entry of P_one[ ][ ] is in the range of 1 to
128, corresponding to a probability of 1/256 to 128/256 of the next error bit (bit E, See Figure 10.5)..."

Fixes: Timeout (42sec ->1sec)
Fixes: 18181/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_DST_fuzzer-5736646250594304
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

diff --git a/libavcodec/dstdec.c b/libavcodec/dstdec.c
index e0519d2a0b9db1a118fbef449627e42e8fc8db1c..ae3fe428ddba0d76c0375b6b5e1bd966530917e4 100644 (file)
--- a/libavcodec/dstdec.c
+++ b/libavcodec/dstdec.c
@@ -162,6 +162,10 @@ static int read_table(GetBitContext *gb, Table *t, const int8_t code_pred_coeff[
                     c -= (x + 4) / 8;
                 else
                     c += (-x + 3) / 8;
+                if (!is_signed) {
+                    if (c < offset || c >= offset + (1<<coeff_bits))
+                        return AVERROR_INVALIDDATA;
+                }
                 t->coeff[i][j] = c;
             }
         }