4 This document explains guidelines that should be observed (or ignored with
5 good reason) when writing filters for libavfilter.
7 In this document, the word “frame” indicates either a video frame or a group
8 of audio samples, as stored in an AVFilterBuffer structure.
14 The query_formats method should set, for each input and each output links,
15 the list of supported formats.
17 For video links, that means pixel format. For audio links, that means
18 channel layout, and sample format (the sample packing is implied by the
21 The lists are not just lists, they are references to shared objects. When
22 the negotiation mechanism computes the intersection of the formats
23 supported at each ends of a link, all references to both lists are
24 replaced with a reference to the intersection. And when a single format is
25 eventually chosen for a link amongst the remaining list, again, all
26 references to the list are updated.
28 That means that if a filter requires that its input and output have the
29 same format amongst a supported list, all it has to do is use a reference
30 to the same list of formats.
33 Buffer references ownership and permissions
34 ===========================================
39 Audio and video data are voluminous; the buffer and buffer reference
40 mechanism is intended to avoid, as much as possible, expensive copies of
41 that data while still allowing the filters to produce correct results.
43 The data is stored in buffers represented by AVFilterBuffer structures.
44 They must not be accessed directly, but through references stored in
45 AVFilterBufferRef structures. Several references can point to the
46 same buffer; the buffer is automatically deallocated once all
47 corresponding references have been destroyed.
49 The characteristics of the data (resolution, sample rate, etc.) are
50 stored in the reference; different references for the same buffer can
51 show different characteristics. In particular, a video reference can
52 point to only a part of a video buffer.
54 A reference is usually obtained as input to the start_frame or
55 filter_samples method or requested using the ff_get_video_buffer or
56 ff_get_audio_buffer functions. A new reference on an existing buffer can
57 be created with the avfilter_ref_buffer. A reference is destroyed using
58 the avfilter_unref_bufferp function.
63 At any time, a reference “belongs” to a particular piece of code,
64 usually a filter. With a few caveats that will be explained below, only
65 that piece of code is allowed to access it. It is also responsible for
66 destroying it, although this is sometimes done automatically (see the
67 section on link reference fields).
69 Here are the (fairly obvious) rules for reference ownership:
71 * A reference received by the start_frame or filter_samples method
72 belong to the corresponding filter.
74 Special exception: for video references: the reference may be used
75 internally for automatic copying and must not be destroyed before
76 end_frame; it can be given away to ff_start_frame.
78 * A reference passed to ff_start_frame or ff_filter_samples is given
79 away and must no longer be used.
81 * A reference created with avfilter_ref_buffer belongs to the code that
84 * A reference obtained with ff_get_video_buffer of ff_get_audio_buffer
85 belongs to the code that requested it.
87 * A reference given as return value by the get_video_buffer or
88 get_audio_buffer method is given away and must no longer be used.
93 The AVFilterLink structure has a few AVFilterBufferRef fields. Here are
94 the rules to handle them:
96 * cur_buf is set before the start_frame and filter_samples methods to
97 the same reference given as argument to the methods and belongs to the
98 destination filter of the link. If it has not been cleared after
99 end_frame or filter_samples, libavfilter will automatically destroy
100 the reference; therefore, any filter that needs to keep the reference
101 for longer must set cur_buf to NULL.
103 * out_buf belongs to the source filter of the link and can be used to
104 store a reference to the buffer that has been sent to the destination.
105 If it is not NULL after end_frame or filter_samples, libavfilter will
106 automatically destroy the reference.
108 If a video input pad does not have a start_frame method, the default
109 method will request a buffer on the first output of the filter, store
110 the reference in out_buf and push a second reference to the output.
112 * src_buf, cur_buf_copy and partial_buf are used by libavfilter
113 internally and must not be accessed by filters.
115 Reference permissions
116 ---------------------
118 The AVFilterBufferRef structure has a perms field that describes what
119 the code that owns the reference is allowed to do to the buffer data.
120 Different references for the same buffer can have different permissions.
122 For video filters, the permissions only apply to the parts of the buffer
123 that have already been covered by the draw_slice method.
125 The value is a binary OR of the following constants:
127 * AV_PERM_READ: the owner can read the buffer data; this is essentially
128 always true and is there for self-documentation.
130 * AV_PERM_WRITE: the owner can modify the buffer data.
132 * AV_PERM_PRESERVE: the owner can rely on the fact that the buffer data
133 will not be modified by previous filters.
135 * AV_PERM_REUSE: the owner can output the buffer several times, without
136 modifying the data in between.
138 * AV_PERM_REUSE2: the owner can output the buffer several times and
139 modify the data in between (useless without the WRITE permissions).
141 * AV_PERM_ALIGN: the owner can access the data using fast operations
142 that require data alignment.
144 The READ, WRITE and PRESERVE permissions are about sharing the same
145 buffer between several filters to avoid expensive copies without them
146 doing conflicting changes on the data.
148 The REUSE and REUSE2 permissions are about special memory for direct
149 rendering. For example a buffer directly allocated in video memory must
150 not modified once it is displayed on screen, or it will cause tearing;
151 it will therefore not have the REUSE2 permission.
153 The ALIGN permission is about extracting part of the buffer, for
154 copy-less padding or cropping for example.
157 References received on input pads are guaranteed to have all the
158 permissions stated in the min_perms field and none of the permissions
159 stated in the rej_perms.
161 References obtained by ff_get_video_buffer and ff_get_video_buffer are
162 guaranteed to have at least all the permissions requested as argument.
164 References created by avfilter_ref_buffer have the same permissions as
165 the original reference minus the ones explicitly masked; the mask is
166 usually ~0 to keep the same permissions.
168 Filters should remove permissions on reference they give to output
169 whenever necessary. It can be automatically done by setting the
170 rej_perms field on the output pad.
172 Here are a few guidelines corresponding to common situations:
174 * Filters that modify and forward their frame (like drawtext) need the
177 * Filters that read their input to produce a new frame on output (like
178 scale) need the READ permission on input and and must request a buffer
179 with the WRITE permission.
181 * Filters that intend to keep a reference after the filtering process
182 is finished (after end_frame or filter_samples returns) must have the
183 PRESERVE permission on it and remove the WRITE permission if they
184 create a new reference to give it away.
186 * Filters that intend to modify a reference they have kept after the end
187 of the filtering process need the REUSE2 permission and must remove
188 the PRESERVE permission if they create a new reference to give it
195 The purpose of these rules is to ensure that frames flow in the filter
196 graph without getting stuck and accumulating somewhere.
198 Simple filters that output one frame for each input frame should not have
201 start_frame / filter_samples
202 ----------------------------
204 These methods are called when a frame is pushed to the filter's input.
205 They can be called at any time except in a reentrant way.
207 If the input frame is enough to produce output, then the filter should
208 push the output frames on the output link immediately.
210 As an exception to the previous rule, if the input frame is enough to
211 produce several output frames, then the filter needs output only at
212 least one per link. The additional frames can be left buffered in the
213 filter; these buffered frames must be flushed immediately if a new input
216 (Example: framerate-doubling filter: start_frame must (1) flush the
217 second copy of the previous frame, if it is still there, (2) push the
218 first copy of the incoming frame, (3) keep the second copy for later.)
220 If the input frame is not enough to produce output, the filter must not
221 call request_frame to get more. It must just process the frame or queue
222 it. The task of requesting more frames is left to the filter's
223 request_frame method or the application.
225 If a filter has several inputs, the filter must be ready for frames
226 arriving randomly on any input. Therefore, any filter with several inputs
227 will most likely require some kind of queuing mechanism. It is perfectly
228 acceptable to have a limited queue and to drop frames when the inputs
234 This method is called when a frame is wanted on an output.
236 For an input, it should directly call start_frame or filter_samples on
237 the corresponding output.
239 For a filter, if there are queued frames already ready, one of these
240 frames should be pushed. If not, the filter should request a frame on
241 one of its inputs, repeatedly until at least one frame has been pushed.
244 if request_frame could produce a frame, it should return 0;
245 if it could not for temporary reasons, it should return AVERROR(EAGAIN);
246 if it could not because there are no more frames, it should return
249 The typical implementation of request_frame for a filter with several
250 inputs will look like that:
256 while (!frame_pushed) {
257 input = input_where_a_frame_is_most_needed();
258 ret = avfilter_request_frame(input);
259 if (ret == AVERROR_EOF) {
260 process_eof_on_input();
261 } else if (ret < 0) {
267 Note that, except for filters that can have queued frames, request_frame
268 does not push frames: it requests them to its input, and as a reaction,
269 the start_frame / filter_samples method will be called and do the work.