swscale: increase yuv2rgb table headroom

Fixes out of array access
Fixes: case2_bad_read_yuv2rgbx32.mp4
Found-by: Michal Zalewski <lcamtuf@coredump.cx>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/mov: fix integer overflow of size

Fixes: case1_call_stack_overflow.mp4
Found-by: Michal Zalewski <lcamtuf@coredump.cx>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/mov: check atom nesting depth

Fixes call stack overflow
Fixes: case1_call_stack_overflow.mp4
Found-by: Michal Zalewski <lcamtuf@coredump.cx>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/utvideodec: Fix handling of slice_height=0

Fixes out of array accesses
Fixes: asan_heap-oob_25bcd7e_3783_cov_3553517262_utvideo_rgba_median.avi
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

doc/filters: fix typos and spelling errors

Prompted-by: Carl Oxwall <@oxwall>
Signed-off-by: Lou Logan <lou@lrcd.com>

avcodec/xface: Add asserts to limit nb_words from becoming too large

Approved-by: Stefano Sabatini <stefasab@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/xface: correct the XFACE_MAX_* values

Fixes out of array access

Fixes: asan_stack-oob_32c12e5_2536_cov_2442316831_lena.xface
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/vmdvideo: Check len before using it in method 3

Fixes out of array access
Fixes: asan_heap-oob_4d23ba_91_cov_3853393937_128.vmd
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/hlsenc: Use av_freep() to avoid leaving stale pointers in memory

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/flvdec: Use av_freep() avoid leaving stale pointers in memory

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/hdsenc: Use av_freep() avoid leaving stale pointers in memory

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

lavfi: add colorlevels filter

Signed-off-by: Paul B Mahol <onemda@gmail.com>

avformat/hlsenc: fix hls_write_trailer() on hls_start() failure

Close segment I/O context and append segment in hls_write_trailer() only
when segment I/O context is allocated.

Signed-off-by: Christian Suloway <csuloway@globaleagleent.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

snow_dwt: Don't try and free members of non-existent arrays

If allocation fails earlier on, and the next frame is processed,
the slice buffer could be left in a state where line and data_stack
have already been freed, or are otherwise null pointers.

Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>

vp9/x86: save one register on 32bit idct32x32.

Fixes build on win32.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit '6f4364aba9d70dc5fd9f1c88b9c03bf9ea893d40'

* commit '6f4364aba9d70dc5fd9f1c88b9c03bf9ea893d40':
  mov: Fix handling of zero-length metadata values

Conflicts:
libavformat/mov.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>

ffmpeg: Check for subtitle_out malloc failure

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/tiffenc: Consistently use ADD_ENTRY1()

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

mov: Fix handling of zero-length metadata values

Since 3cec81f4d4, a zero-length metadata value would try to
allocate 2*0 bytes, where av_malloc() returns NULL.

Always add one to the allocated length, to allow space for
a null terminator in the zero-length case.

Incidentally, this fixes fate-alac on RVCT 4.0, where a compiler
bug seems to mess up the mov muxer to the point that it writes
the wrong sort of metadata. Previously this bug was undetected,
but since 3cec81f4d4 such mov files started returning
AVERROR(ENOMEM) in the mov demuxer.

Signed-off-by: Martin Storsjö <martin@martin.st>

Merge commit '10d4c5e55e5b23e165aa4c7723073ebe2c2e7da0'

* commit '10d4c5e55e5b23e165aa4c7723073ebe2c2e7da0':
  tiff: set the correct return value when check_size() fails

Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit '3c27275c1309190f2d6ed69140b67d014215b6c9'

* commit '3c27275c1309190f2d6ed69140b67d014215b6c9':
  tiff: Check the check_size() return value and forward it

Conflicts:
libavcodec/tiffenc.c

See: d50aa006fb3430bedc3872ba10e028a714499625
Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit '4690e01c3aaf495c87127e5dc74aa347197dbc0b'

* commit '4690e01c3aaf495c87127e5dc74aa347197dbc0b':
  prores: Evaluate all the quantizers

Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit '490a3ebf36821b81f73e34ad3f554cb523dd2682'

* commit '490a3ebf36821b81f73e34ad3f554cb523dd2682':
  matroskadec: Fix read-after-free in matroska_read_seek()

See: 33301f001747d7a542073c634cc81da5eff051cf
Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit 'd75190aa93206c02cb4f18a66d2c927d6a0198e1'

* commit 'd75190aa93206c02cb4f18a66d2c927d6a0198e1':
  mpegvideo: move REBASE_PICTURE where it is used

Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit '63fcedcf01247f15b72c0d1f959bb3b612853ba8'

* commit '63fcedcf01247f15b72c0d1f959bb3b612853ba8':
  mpegvideo: remove unused function declaration

Merged-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/hevc: Print an error if luma_log2_weight_denom is out of range

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit 'b2bece5e9a4dd96142962c6798e187c77b88bbfa'

* commit 'b2bece5e9a4dd96142962c6798e187c77b88bbfa':
  hevc: always clip luma_log2_weight_denom

Merged-by: Michael Niedermayer <michaelni@gmx.at>

lavf/concatdec: handle NOPTS start_time.

Fix trac ticket #3598.

Signed-off-by: Nicolas George <george@nsup.org>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Changelog: Add 10bit spp

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avfilter/vf_spp: support 10bit per sample

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/avdct: support >8bit per sample dcts

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

doc/ffplay: remove duplicate threads option, this is documented in codecs.*

Fixes Ticket3610

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/xmv: Check return code of ff_alloc_extradata()

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/flic: se av_freep() to avoid leaving stale extradata pointer

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/rtpdec_mpeg4: Use av_freep() to avoid leaving stale extradata pointer

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/xmv: use av_freep() to avoid leaving stale extradata pointer

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

tiff: set the correct return value when check_size() fails

Only one instance affected and solved as other occurences.

tiff: Check the check_size() return value and forward it

Also use the same type for add_entry and check_size.

Bug-Id: CID 700699
CC: libav-stable@libav.org
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Signed-off-by: Vittorio Giovara <vittorio.giovarao@gmail.com>

prores: Evaluate all the quantizers

Prevent an uninitialized data access.

CC: libav-stable@libav.org
Bug-Id: CID 703824 / CID 703825
Signed-off-by: Vittorio Giovara <vittorio.giovara@gmail.com>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

matroskadec: Fix read-after-free in matroska_read_seek()

In matroska_read_seek(), |tracks| is assigned at the begining of the
function. However, functions like matroska_parse_cues() could reallocate
the tracks and invalidate |tracks|.

This assigns |tracks| only before using it, so that it will not get
invalidated elsewhere.

Bug-Id: chromium/427266

mpegvideo: move REBASE_PICTURE where it is used

Drop an unused #undef from h264 decoder.

mpegvideo: remove unused function declaration

hevc: always clip luma_log2_weight_denom

Its value shall be between 0 and 7 according to the specifications.

CC: libav-stable@libav.org
Bug-Id: CID 1257502

avformat/assenc: Use av_freep() to avoid leaving stale pointers in memory

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/avidec: Use av_freep() to avoid leaving stale pointers in memory

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

configure: create the tests directory like the doc directory

This fixes an issue where the tests directory is not created for out of tree
builds before its needed

Tested-by: Dave Yeo <daveryeo@telus.net>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit 'bab8d73765238bfeea7a8dfd454f40ac213b3d22'

* commit 'bab8d73765238bfeea7a8dfd454f40ac213b3d22':
  vaapi: wrap codec specific functions in appropiate #ifs

Conflicts:
libavcodec/vaapi.c

See: f2118d44c9f18b3245bec2415c563d94201dcf5a
Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit '5a82ad644f281701eb22e0359fd732fac67aafa4'

* commit '5a82ad644f281701eb22e0359fd732fac67aafa4':
  svq1dec: Unbreak the scratch buffer allocation

Conflicts:
libavcodec/svq1dec.c

See: 4213fc5b9eebec53c7d22b770c3f1ceecca1c113
Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit 'bcbdeb318e0828cf00b458885ffa1e1596c85d06'

* commit 'bcbdeb318e0828cf00b458885ffa1e1596c85d06':
  asvenc: Forward the error instead of asserting on it

Conflicts:
libavcodec/asvenc.c

Not merged, conditions which are impossible should be checked by assert*()
adding impossible to be true error checks has the potential to confuse

Merged-by: Michael Niedermayer <michaelni@gmx.at>

lavu/frame: fix malloc error path in av_frame_copy_props()

The error path frees all side data, but forgets to reset the side data
count. This can blow up later in av_frame_unref() and free_side_data().

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

configure: Set the default for --shlibdir to --libdir.

Fixes ticket #4183.

Reviewed-by: Ivan Kalvachev

vp9/x86: sse2 MC assembly.

Also a slight change to the ssse3 code, which prevents a theoretical
overflow in the sharp filter.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

lavf/mxfdec: Fix memleaks reading corrupt files.

Fixes ticket #4173.

Reviewed-by: Tomas Härdin

lavf/mxfdec: Factorize mxf_read_close().

This will allow to fix leaks in a subsequent commit.

vp9/x86: 32bit and sse2 support for vp9 inverse transform assembly

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/dv: Use av_freep() to avoid leaving stale pointers in memory

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

cmdutils: use macros for device test

Signed-off-by: Lukasz Marek <lukasz.m.luki2@gmail.com>

lavd/alsa-audio-common: mark default device in device list

Signed-off-by: Lukasz Marek <lukasz.m.luki2@gmail.com>

lavd/avdevice: use better option types for caps options

Using dedicated types allows to use format/layout names,
not just raw int values.

Signed-off-by: Lukasz Marek <lukasz.m.luki2@gmail.com>

Merge commit 'fd665f7f48fa7db89eb9a93ac33919f6adc40f9d'

* commit 'fd665f7f48fa7db89eb9a93ac33919f6adc40f9d':
  avconv: Use the mpeg12 private option scan_offset

Conflicts:
ffmpeg_opt.c

See: e479013ae41050ad9b76429d8c6ab70684cddc93
Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit '559fa0d41b5cf8f7e1ba89d1ac70e74712eb2816'

* commit '559fa0d41b5cf8f7e1ba89d1ac70e74712eb2816':
  vdpau: add mapping for H.264 Constrained Baseline profile and fallback

Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit '8502c1e9ff9c1dbb6e467630c048d098f4064021'

* commit '8502c1e9ff9c1dbb6e467630c048d098f4064021':
  vdpau: add mapping for H.264 Extended profile

Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit '8b51bcfed75721e0d20832e5a90e1d585318c9d9'

* commit '8b51bcfed75721e0d20832e5a90e1d585318c9d9':
  vdpau: revector macro to reduce line span

Merged-by: Michael Niedermayer <michaelni@gmx.at>

Merge commit 'e10e6651b50b4087f08813f3e503620db7413d3a'

* commit 'e10e6651b50b4087f08813f3e503620db7413d3a':
  thread: use "" instead of <> for including the w32pthreads wrapper

See: 32eadfe453c32788ea57968e6e80f673f434739d
Merged-by: Michael Niedermayer <michaelni@gmx.at>

vaapi: wrap codec specific functions in appropiate #ifs

Fix linking when only a subset of vaapi decoders is enabled.

Bug-Id: 760
CC: libav-stable@libav.org
Signed-off-by: Rémi Denis-Courmont <remi@remlab.net>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

svq1dec: Unbreak the scratch buffer allocation

The input packets are always assumed to be padded and
the av_fast_ family of function takes a pointer to a pointer.

Thanks to Nicolas Dufresne <nicolas.dufresne@collabora.com> for
a similar patch.

Introduced in 7b588bb691644e1b3c168b99accf74248a24e3cf.

Bug-Id: 766
CC: libav-stable@libav.org
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

asvenc: Forward the error instead of asserting on it

Bug-Id: CID 732997
CC: libav-stable@libav.org

avformat/utils: Do not update programs streams from program-less streams in update_wrap_reference()

Fixes Ticket3686

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/mp3dec: Name the dummy variable as what it is, to avoid confusion

Found-by: nattyzs <nattyzs@163.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avconv: Use the mpeg12 private option scan_offset

Introduced in aed790070486b1b01b48106310d9d0ca1730e459

Bug-Id: debian/773055
CC: libav-stable@libav.org
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Signed-off-by: Anton Khirnov <anton@khirnov.net>

vdpau: add mapping for H.264 Constrained Baseline profile and fallback

Old VDPAU drivers do not support this newly defined profile, so falling
back to Main profile is necessary for backward binary compatibility.

Signed-off-by: Anton Khirnov <anton@khirnov.net>

vdpau: add mapping for H.264 Extended profile

Signed-off-by: Anton Khirnov <anton@khirnov.net>

vdpau: revector macro to reduce line span

Signed-off-by: Anton Khirnov <anton@khirnov.net>

thread: use "" instead of <> for including the w32pthreads wrapper

Found-by: Dave Yeo <dave.r.yeo@gmail.com>

avformat/aviobuf: Check that avio_seek() target is non negative

Fixes out of array access

Suggested-by: Andrew Scherkus <scherkus@google.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avfilter/vf_dctdnoiz: use av_freep() to avoid leaving stale pointers in memory

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avfilter/f_sendcmd: Use av_freep() to avoid leaving stale pointers in memory

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/xiph: mark returned header pointers const from avpriv_split_xiph_headers()

Reviewed-by: Reimar Döffinger <Reimar.Doeffinger@gmx.de>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/nutenc: store the actual r_frame_rate in the corresponding field if available

Fixes Ticket 3629 part3

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

ffmpeg: forward r_frame_rate to the muxer in case of stream copy

Fixes Ticket3629 part2

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/utils: use r_frame_rate only in the demuxer for ff_compute_frame_duration()

This is primarly to maintain current behavior when r_frame_rate is set for muxers
and could be reverted if it has some advantage

Fixes Ticket3629 part1

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec: fix wording in AVSubtitleRect doxy

swresample/soxr_resample: fix error handling

Fixes CID1257659

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

fix "than" typo

Found-by reimar
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/utils: Do not estimate the container bitrate from the streams if one video stream lacks the bitrate

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avfilter/vf_uspp: fix gray pixfmt handling

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avfilter/vf_uspp: fix used chroma subsampling factors

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avfilter/vf_uspp: fix used pix_fmt

Found-by: carl
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/dvdsubdec: Avoid strerror() due to thread saftey

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avcodec/libxvid_rc: Avoid strerror() for thread saftey

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/mov: Simplify code by using OFFSET() and FLAGS macros

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avfilter/vf_kerndeint: use av_freep(), avoid leaving stale pointers in memory

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avfilter/vf_unsharp: use av_freep() avoid leaving stale pointers in memory

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avfilter/vf_uspp: Add grayscale support

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avfilter/vf_uspp: assert that the qp_type is valid

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

doc: add FFREPORT example

And fix a few nits.

Signed-off-by: Lou Logan <lou@lrcd.com>

lavu/dict: check for malloc failures

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

avformat/utils: check for malloc failure

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

lavc/avpacket: check for malloc failure

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Delete mp=uspp