libavformat: Check mkdir return error codes

libavformat: Check mkdir return error codes

Previously, the returned error codes were intentionally ignored
(see fadd3a68213), to avoid aborting if the directory already
existed. If the mkdir actually failed, this was caught when
opening files within the directory fails anyway.

By handling the error code here (but explicitly ignoring EEXIST),
the error messages and return codes in these cases are more
appropriate and less confusing.

Signed-off-by: Martin Storsjö <martin@martin.st>