From: Steinar H. Gunderson <sgunderson@bigfoot.com>
Date: Sun, 16 Apr 2023 08:35:13 +0000 (+0200)
Subject: Improve selection of software formats on hwaccel fallback.
X-Git-Tag: 2.2.1~3
X-Git-Url: https://git.sesse.net/?p=nageru;a=commitdiff_plain;h=cb1dcf409cde897b011e9d9d3cd3cf2974956190

Improve selection of software formats on hwaccel fallback.

We assumed the first format was a software format, whereas in practice,
it would now seemingly be vdpau or cuda, causing multiple trips
through the selection function.
---

diff --git a/nageru/ffmpeg_capture.cpp b/nageru/ffmpeg_capture.cpp
index e099d18..5cf1d8c 100644
--- a/nageru/ffmpeg_capture.cpp
+++ b/nageru/ffmpeg_capture.cpp
@@ -503,7 +503,15 @@ AVPixelFormat get_hw_format(AVCodecContext *ctx, const AVPixelFormat *fmt)
 		fprintf(stderr, "Decoder '%s' does not support device type '%s'.\n", ctx->codec->name, av_hwdevice_get_type_name(type));
 	}
 
-	// We found no VA-API formats, so take the best software format.
+	// We found no VA-API formats, so take the first software format.
+	for (const AVPixelFormat *fmt_ptr = fmt; *fmt_ptr != -1; ++fmt_ptr) {
+		if ((av_pix_fmt_desc_get(*fmt_ptr)->flags & AV_PIX_FMT_FLAG_HWACCEL) == 0) {
+			fprintf(stderr, "Falling back to software format %s.\n", av_get_pix_fmt_name(*fmt_ptr));
+			return *fmt_ptr;
+		}
+	}
+
+	// Fallback: Just return anything. (Should never really happen.)
 	return fmt[0];
 }