avcodec/mpeg12dec: Sanitize start codes earlier

The MPEG-1/2 decoder uses avpriv_find_start_code to search for start
codes and worked with the resulting start code before checking that it
is really a start code of a slice. In particular, if the picture is so
big that a slice_vertical_position_extension is present, it added the
slice_vertical_position_extension as if it had a slice. Then a left
shift is performed, without making sure that the value to be shifted is
nonnegative.
Afterwards the end result is checked, but even if a start code of a
non-slice has been found, it might pass these checks: If
slice_vertical_position_extension is present a start code <
SLICE_MIN_START_CODE can lead to a macroblock-row index that appears
valid. Furthermore, the left shift might make an invalid start code
appear valid by discarding the highest bit.
This has been fixed by checking directly after avpriv_find_start_code
has returned.

Fixes ticket #8162 (which is about the undefined left shifts).

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

diff --git a/libavcodec/mpeg12dec.c b/libavcodec/mpeg12dec.c
index e8bdbdb61aa79540f3d5da7ac6036475820301eb..775579f9f0b36febcc9873244006975a82ed494b 100644 (file)
--- a/libavcodec/mpeg12dec.c
+++ b/libavcodec/mpeg12dec.c
@@ -2016,13 +2016,15 @@ static int slice_decode_thread(AVCodecContext *c, void *arg)
 
         start_code = -1;
         buf        = avpriv_find_start_code(buf, s->gb.buffer_end, &start_code);
+        if (start_code < SLICE_MIN_START_CODE || start_code > SLICE_MAX_START_CODE)
+            return AVERROR_INVALIDDATA;
         mb_y       = start_code - SLICE_MIN_START_CODE;
         if (s->codec_id != AV_CODEC_ID_MPEG1VIDEO && s->mb_height > 2800/16)
             mb_y += (*buf&0xE0)<<2;
         mb_y <<= field_pic;
         if (s->picture_structure == PICT_BOTTOM_FIELD)
             mb_y++;
-        if (mb_y < 0 || mb_y >= s->end_mb_y)
+        if (mb_y >= s->end_mb_y)
             return AVERROR_INVALIDDATA;
     }
 }