nutdec: reject negative value_len in read_sm_data

If it is negative, it can cause the byte position to move backwards in
avio_skip, which in turn makes sm_size negative and thus size larger
than the size of the packet buffer, causing invalid writes in avio_read.

Also fix potential overflow of avio_tell(bc) + value_len.

Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>

diff --git a/libavformat/nutdec.c b/libavformat/nutdec.c
index ef08ad9908bed12874b6b5ce6d9cd44b0511d413..4df6a57a6720b143f43abf071f466bb52c5b7d79 100644 (file)
--- a/libavformat/nutdec.c
+++ b/libavformat/nutdec.c
@@ -934,7 +934,7 @@ static int read_sm_data(AVFormatContext *s, AVIOContext *bc, AVPacket *pkt, int
                 return ret;
             }
             value_len = ffio_read_varlen(bc);
-            if (avio_tell(bc) + value_len >= maxpos)
+            if (value_len < 0 || value_len >= maxpos - avio_tell(bc))
                 return AVERROR_INVALIDDATA;
             if (!strcmp(name, "Palette")) {
                 dst = av_packet_new_side_data(pkt, AV_PKT_DATA_PALETTE, value_len);