avcodec/magicyuv: Fix edge case of building Huffman table

The MagicYUV format stores Huffman tables in its bitstream by coding
the length of a given symbol; it does not code the actual code directly,
instead this is to be inferred by the rule that a symbol is to the left
of every shorter symbol in the Huffman tree and that for symbols of the
same length the symbol is ascending from left to right. With one
exception, this is also what our decoder did.

The exception only matters when there are codes of length 32, because
in this case the first symbol of this length did not get the code 0,
but 1; e.g. if there were exactly two nodes of length 32, then they
would get assigned the codes 1 and 2 and a node of length 31 will get
the 31-bit code 1 which is a prefix of the 32 bit code 2, making the
Huffman table invalid. On the other hand, if there were only one symbol
with the length 32, the earlier code would accept this un-Huffman-tree.

Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

diff --git a/libavcodec/magicyuv.c b/libavcodec/magicyuv.c
index 1b3f4cfc6b92a7c859d68c938c0bb04ffc02853e..17dea69d76fc4795023dca098c79891e2abf580f 100644 (file)
--- a/libavcodec/magicyuv.c
+++ b/libavcodec/magicyuv.c
@@ -86,7 +86,7 @@ static int huff_build(HuffEntry he[], VLC *vlc, int nb_elems)
 
     AV_QSORT(he, nb_elems, HuffEntry, huff_cmp_len);
 
-    code = 1;
+    code = 0;
     for (unsigned i = 0; i < nb_elems; i++) {
         he[i].code = code >> (32 - he[i].len);
         code += 0x80000000u >> (he[i].len - 1);