avformat/mpegenc: Fix memleaks and return values

avformat/mpegenc: Fix memleaks and return values

If there is an error in mpeg_mux_init() (the write_header function of
the various MPEG-PS muxers), two things might happen:

1. Several fifos might leak. Instead of freeing them, the goto fail part
of the functions freed the private data of the AVStreams instead,
although this will be freed later in free_stream() anyway.
2. And if the function is exited via goto fail, it automatically
returned AVERROR(ENOMEM), although this is also used when the error is
not a memory allocation failure.

Both of these issues happened in ticket #8284 and have been fixed.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>