avutil/video_enc_params: Combine overflow checks

avutil/video_enc_params: Combine overflow checks

This patch also fixes a -Wtautological-constant-out-of-range-compare
warning from Clang and a -Wtype-limits warning from GCC on systems
where size_t is 64bits and unsigned 32bits. The reason for this seems
to be that variable (whose value derives from sizeof() and can therefore
be known at compile-time) is used instead of using sizeof() directly in
the comparison.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>