avcodec/omx: fix xFramerate calculation

Integer overflow in the Q16 framerate calculation was sending
invalid values to the OMX encoder.

On the RPI4, this manifested as bitrate controls being ignored
on video streams with 60000/1001 framerates. Video streams with
30000/1001 framerates were not affected.

Signed-off-by: Aman Gupta <aman@tmm1.net>

diff --git a/libavcodec/omx.c b/libavcodec/omx.c
index 837f5df666ae1cd44868c89b5c191821eb1192ea..0a6a3083090e2879d8ec2c6885b70f4a8587c49d 100644 (file)
--- a/libavcodec/omx.c
+++ b/libavcodec/omx.c
@@ -473,9 +473,9 @@ static av_cold int omx_component_init(AVCodecContext *avctx, const char *role)
     in_port_params.format.video.nFrameWidth  = avctx->width;
     in_port_params.format.video.nFrameHeight = avctx->height;
     if (avctx->framerate.den > 0 && avctx->framerate.num > 0)
-        in_port_params.format.video.xFramerate = (1 << 16) * avctx->framerate.num / avctx->framerate.den;
+        in_port_params.format.video.xFramerate = (1LL << 16) * avctx->framerate.num / avctx->framerate.den;
     else
-        in_port_params.format.video.xFramerate = (1 << 16) * avctx->time_base.den / avctx->time_base.num;
+        in_port_params.format.video.xFramerate = (1LL << 16) * avctx->time_base.den / avctx->time_base.num;
 
     err = OMX_SetParameter(s->handle, OMX_IndexParamPortDefinition, &in_port_params);
     CHECK(err);