+Framework Documentation
+
+Copyright (C) 2004-2009 Ushodaya Enterprises Limited
+Author: Charles Yates <charles.yates@pandora.be>
+Last Revision: 2005-05-08
+
+
MLT FRAMEWORK
-------------
it provides a pluggable architecture for the inclusion of new audio/video
sources, filters, transitions and playback devices.
- The MLT framework provides the structure and utility functionality on which
+ The framework provides the structure and utility functionality on which
all of the MLT applications and services are defined.
On its own, the framework provides little more than 'abstract classes' and
This document is split roughly into 3 sections. The first section provides a
basic overview of MLT, the second section shows how it's used and the final
- section shows shows structure and design, with an emphasis on how the system
- is extended.
+ section shows structure and design, with an emphasis on how the system is
+ extended.
Target Audience:
opposed to the implementation details.
It is not required reading for the MLT client/server integration - please
- refer to valerie.txt and dvcp.txt for more details on this area.
+ refer to libmvsp.txt and mvsp.txt for more details on this area.
SECTION 1 - BASIC OVERVIEW
A typical consumer requests MLT Frame objects from the producer, does
something with them and when finished with a frame, closes it.
- /\ A common confusion with the producer/consumer terminoligy used here is
+ /\ A common confusion with the producer/consumer terminology used here is
/!!\ that a consumer may 'produce' something. For example, the libdv consumer
\!!/ produces DV and the libdv producer seems to consume DV. However, the
\/ naming conventions refer only to producers and consumers of MLT Frames.
The defaults requested here are a special case - the NULL usage requests
that we use the default producers and consumers.
- The default producer is "fezzik". This producer matches file names to
+ The default producer is "loader". This producer matches file names to
locate a service to use and attaches 'normalising filters' (such as scalers,
deinterlacers, resamplers and field normalisers) to the loaded content -
these filters ensure that the consumer gets what it asks for.
- The default consumer is "sdl". The combination of fezzik and sdl will
+ The default consumer is "sdl". The combination of loader and sdl will
provide a media player.
In this example, we connect the producer and then start the consumer. We
Also note, you can override the defaults as follows:
- $ MLT_CONSUMER=westley ./hello file.avi
+ $ MLT_CONSUMER=xml ./hello file.avi
- This will create a westley xml document on stdout.
+ This will create a XML document on stdout.
- $ MLT_CONSUMER=westley MLT_PRODUCER=avformat ./hello file.avi
+ $ MLT_CONSUMER=xml MLT_PRODUCER=avformat ./hello file.avi
This will play the video using the avformat producer directly, thus it will
bypass the normalising functions.
+------------------+------------------------------------+------------------+
|MLT_NORMALISATION |The normalisation of the system |PAL or NTSC |
+------------------+------------------------------------+------------------+
- |MLT_PRODUCER |The default producer |"fezzik" or other |
+ |MLT_PRODUCER |The default producer |"loader" or other |
+------------------+------------------------------------+------------------+
|MLT_CONSUMER |The default consumer |"sdl" or other |
+------------------+------------------------------------+------------------+
+ |MLT_TEST_CARD |The default test card producer |any producer |
+ +------------------+------------------------------------+------------------+
These values are initialised from the environment variables of the same
name.
Let's assume that we're adapting the Hello World example, and wish to queue
a number of files for playout, ie:
- hello *.avi
+ hello *.avi
Instead of invoking mlt_factory_producer directly, we'll create a new
function called create_playlist. This function is responsible for creating
- the playlist, creating each producer, appending to the playlist and ensuring
- that all the producers are cleaned up when the playlist is destroyed. The
- last point is important - a close on the playlist won't explicitly these
- producers. In this example, we use unique "data" properties with destructors
- to ensure closing.
+ the playlist, creating each producer and appending to the playlist.
mlt_producer create_playlist( int argc, char **argv )
{
int i = 0;
for ( i = 1; i < argc; i ++ )
{
- // Definie the unique key
- char key[ 256 ];
-
// Create the producer
mlt_producer producer = mlt_factory_producer( NULL, argv[ i ] );
// Add it to the playlist
mlt_playlist_append( playlist, producer );
- // Create a unique key for this producer
- sprintf( key, "producer%d", i );
-
- // Now we need to ensure the producers are destroyed
- mlt_properties_set_data( properties, key, producer, 0, ( mlt_destructor )mlt_producer_close, NULL );
+ // Close the producer (see below)
+ mlt_producer_close( producer );
}
// Return the playlist as a producer
return mlt_playlist_producer( playlist );
}
+ Notice that we close the producer after the append. Actually, what we're
+ doing is closing our reference to it - the playlist creates its own reference
+ to the producer on append and insert, and it will close its reference
+ when the playlist is destroyed[*].
+
+ Note also that if you append multiple instances of the same producer, it
+ will create multiple references to it.
+
Now all we need do is to replace these lines in the main function:
// Create a normalised producer
and we have a means to play multiple clips.
+ [*] This reference functionality was introduced in mlt 0.1.2 - it is 100%
+ compatable with the early mechanism of registering the reference and
+ destructor with the properties of the playlist object.
+
Filters:
properties can be set as needed.
The additional argument in the filter connection is an important one as it
- dicates the 'track' on which the filter operates. For basic producers and
+ dictates the 'track' on which the filter operates. For basic producers and
playlists, there's only one track (0), and as you will see in the next
section, even multiple tracks have a single track output.
+Attached Filters:
+
+ All services can have attached filters.
+
+ Consider the following example:
+
+ // Create a producer
+ mlt_producer producer = mlt_factory_producer( NULL, clip );
+
+ // Get the service object of the producer
+ mlt_producer service = mlt_producer_service( producer );
+
+ // Create a filter
+ mlt_filter filter = mlt_factory_filter( "greyscale" );
+
+ // Create a playlist
+ mlt_playlist playlist = mlt_playlist_init( );
+
+ // Attach the filter to the producer
+ mlt_service_attach( producer, filter );
+
+ // Construct a playlist with various cuts from the producer
+ mlt_playlist_append_io( producer, 0, 99 );
+ mlt_playlist_append_io( producer, 450, 499 );
+ mlt_playlist_append_io( producer, 200, 399 );
+
+ // We can close the producer and filter now
+ mlt_producer_close( producer );
+ mlt_filter_close( filter );
+
+ When this is played out, the greyscale filter will be executed for each frame
+ in the playlist which comes from that producer.
+
+ Further, each cut can have their own filters attached which are executed after
+ the producer's filters. As an example:
+
+ // Create a new filter
+ filter = mlt_factory_filter( "invert", NULL );
+
+ // Get the second 'clip' in the playlist
+ producer = mlt_playlist_get_clip( 1 );
+
+ // Get the service object of the clip
+ service = mlt_producer_service( producer );
+
+ // Attach the filter
+ mlt_service_attach( producer, filter );
+
+ // Close the filter
+ mlt_filter_close( filter );
+
+ Even the playlist itself can have an attached filter:
+
+ // Create a new filter
+ filter = mlt_factory_filter( "watermark", "+Hello.txt" );
+
+ // Get the service object of the playlist
+ service = mlt_playlist_service( playlist );
+
+ // Attach the filter
+ mlt_service_attach( service, filter );
+
+ // Close the filter
+ mlt_filter_close( filter );
+
+ And, of course, the playlist, being a producer, can be cut up and placed on
+ another playlist, and filters can be attached to those cuts or on the new
+ playlist itself and so on ad nauseum.
+
+ The main advantage of attached filters is that they remain attached and don't
+ suffer from the maintenance problems associated with items being inserted and
+ displacing calculated in/out points - this being a major issue if you
+ exclusively use the connect or insert detached filters in a multitrack field
+ (described below).
+
+
+Introducing the Mix:
+
+ The mix is the simplest way to introduce transitions between adjacent clips
+ on a playlist.
+
+ Consider the following playlist:
+
+ +-+----------------------+----------------------------+-+
+ |X|A |B |X|
+ +-+----------------------+----------------------------+-+
+
+ Let's assume that the 'X' is a 'black clip' of 50 frames long.
+
+ When you play this out, you'll get a 50 frames of black, abrupt cut into
+ A, followed by an abrupt cut into B, and finally into black again.
+
+ The intention is to convert this playlist into something like:
+
+ +-+---------------------+-+------------------------+-+
+ |X|A |A|B |B|
+ |A| |B| |X|
+ +-+---------------------+-+------------------------+-+
+
+ Where the clips which refer to 2 clips represent a transition. Notice that
+ the representation of the second playlist is shorter than the first - this is
+ to be expected - a single transition of 50 frames between two clips will
+ reduce the playtime of the result by 50 frames.
+
+ This is done via the use of the mlt_playlist_mix method. So, assuming you get
+ a playlist as shown in the original diagram, to do the first mix, you could do
+ something like:
+
+ // Create a transition
+ mlt_transition transition = mlt_factor_transition( "luma", NULL );
+
+ // Mix the first and second clips for 50
+ mlt_playlist_mix( playlist, 0, 50, transition );
+
+ // Close the transition
+ mlt_transition_close( transition );
+
+ This would give you the first transition, subsequently, you would apply a similar
+ technique to mix clips 1 and 2. Note that this would create a new clip on the
+ playlist, so the next mix would be between 3 and 4.
+
+ As a general hint, to simplify the requirement to know the next clip index,
+ you might find the following simpler:
+
+ // Get the number of clips on the playlist
+ int i = mlt_playlist_count( );
+
+ // Iterate through them in reverse order
+ while ( i -- )
+ {
+ // Create a transition
+ mlt_transition transition = mlt_factor_transition( "luma", NULL );
+
+ // Mix the first and second clips for 50
+ mlt_playlist_mix( playlist, i, 50, transition );
+
+ // Close the transition
+ mlt_transition_close( transition );
+ }
+
+ There are other techniques, like using the mlt_playlist_join between the
+ current clip and the newly created one (you can determine if a new clip was
+ created by comparing the playlist length before and after the mix call).
+
+ Internally, the mlt_playlist_mix call generates a tractor and multitrack as
+ described below. Like the attached filters, the mix makes life very simple
+ when you're inserting items into the playlist.
+
+ Also note that it allows a simpler user interface - instead of enforcing the
+ use of a complex multitrack object, you can do many operations on a single
+ track. Thus, additional tracks can be used to introduce audio dubs, mixes
+ or composites which are independently positioned and aren't affected by
+ manipulations on other tracks. But hey, if you want a bombastic, confusing
+ and ultimately frustrating traditional NLE experience, that functionality
+ is provided too ;-).
+
+
+Practicalities and Optimisations:
+
+ In the previous two sections I've introduced some powerful functionality
+ designed to simplify MLT usage. However, a general issue comes into this -
+ what happens when you introduce a transition between two cuts from the same
+ bit of video footage?
+
+ Anyone who is familiar with video compression will be aware that seeking
+ isn't always without consequence from a performance point of view. So if
+ you happen to require two frames from the same clip for a transition, the
+ processing is going to be excessive and the result will undoubtedly be very
+ unpleasant, especially if you're rendering in realtime...
+
+ So how do we get round this?
+
+ Actually, it's very simple - you invoke mlt_producer_optimise on the top
+ level object after a modification and MLT will determine how to handle it.
+ Internally, it determines the maximum number of overlapping instances
+ throughout the object and creates clones and assigns clone indexes as
+ required.
+
+ In the mix example above, you can simply call:
+
+ // Optimise the playlist
+ mlt_producer_optimise( mlt_playlist_producer( playlist ) );
+
+ after the mix calls have be done. Note that this is automatically applied
+ to deserialised MLT XML.
+
+
Multiple Tracks and Transitions:
- MLT's approach to mutliple tracks is governed by two requirements:
+ MLT's approach to multiple tracks is governed by two requirements:
1) The need for a consumer and producer to communicate with one another via
a single frame;
+------------------------------+
The overlapping areas of track 0 and 1 would (presumably) have some kind of
- transition - without a transition, the frames from a1 and a2 would be shown
- during the areas of overlap.
+ transition - without a transition, the frames from b1 and b2 would be shown
+ during the areas of overlap (ie: by default, the higher numbered track takes
+ precedence over the lower numbered track).
MLT has a multitrack object, but it is not a producer in the sense that it
can be connected directly to a consumer and everything will work correctly.
evenly, the correct frame is output and that we have 'producer like'
behaviour.
- Thus, a mulitrack is conceptually 'pulled' by a tractor as shown here:
+ Thus, a multitrack is conceptually 'pulled' by a tractor as shown here:
+----------+
|multitrack|
+----------+
With a combination of the two, we can now connect multitracks to consumers.
- The first non-test card will be retrieved and passed on.
+ The last non-test card will be retrieved and passed on.
The tracks can be producers, playlists, or even other tractors.
- Now we wish to insert filters and transitions between the mulitrack and the
+ Now we wish to insert filters and transitions between the multitrack and the
tractor. We can do this directly by inserting filters directly between the
tractor and the multitrack, but this involves a lot of connecting and
reconnecting left and right producers and consumers, and it seemed only fair
that we should be able to automate that process.
-
+
So in keeping with our agricultural theme, the concept of the 'field' was
born. We 'plant' filters and transitions in the field and the tractor pulls
the multitrack (think of a combine harvester :-)) over the field and
| +------+ | +-------------+ +-------+
+----------+
- In reality, we create a field first, and from that we obtain a multitrack
- and a tractor. We can then populate the multitrack, field and finally,
- connect the tractor to the consumer.
-
- The reasoning behind this is possibly flawed - it might have made more
- sense to produce the tractor and have it encapsulate the field and the
- multitrack as that is how it looks to a connected consumer:
+ So, we need to create the tractor first, and from that we obtain the
+ multitrack and field objects. We can populate these and finally
+ connect the tractor to a consumer.
+
+ In essence, this is how it looks to the consumer:
+-----------------------------------------------+
|tractor +--------------------------+ |
mlt_producer create_tracks( int argc, char **argv )
{
- // Create the field
- mlt_field field = mlt_field_init( );
+ // Create the tractor
+ mlt_tractor tractor = mlt_tractor_new( );
+
+ // Obtain the field
+ mlt_field field = mlt_tractor_field( tractor );
// Obtain the multitrack
- mlt_multitrack multitrack = mlt_field_multitrack( field );
-
- // Obtain the tractor
- mlt_tractor tractor = mlt_field_tractor( field );
+ mlt_multitrack multitrack = mlt_tractor_multitrack( tractor );
- // Obtain a composite transition
- mlt_transition transition = mlt_factory_transition( "composite", "10%,10%:15%x15%" );
+ // Create a composite transition
+ mlt_transition transition = mlt_factory_transition( "composite", "10%/10%:15%x15%" );
// Create track 0
mlt_producer track0 = create_playlist( argc, argv );
- // Create the watermark track - note we NEED fezzik for scaling here
- mlt_producer track1 = mlt_factory_producer( "fezzik", "pango" );
+ // Create the watermark track - note we NEED loader for scaling here
+ mlt_producer track1 = mlt_factory_producer( "loader", "pango" );
// Get the length of track0
mlt_position length = mlt_producer_get_playtime( track0 );
mlt_properties_set_position( properties, "in", 0 );
mlt_properties_set_position( properties, "out", length - 1 );
mlt_properties_set_position( properties, "length", length );
-
+ mlt_properties_set_int( properties, "a_track", 0 );
+ mlt_properties_set_int( properties, "b_track", 1 );
+
// Now set the properties on the transition
properties = mlt_transition_properties( transition );
mlt_properties_set_position( properties, "in", 0 );
// Now plant the transition
mlt_field_plant_transition( field, transition, 0, 1 );
-
- // Now set the properties on the tractor
- properties = mlt_tractor_properties( tractor );
- mlt_properties_set_data( properties, "multitrack", multitrack, 0, ( mlt_destructor )mlt_multitrack_close, NULL );
- mlt_properties_set_data( properties, "field", field, 0, ( mlt_destructor )mlt_field_close, NULL );
- mlt_properties_set_data( properties, "track0", track0, 0, ( mlt_destructor )mlt_producer_close, NULL );
- mlt_properties_set_data( properties, "track1", track1, 0, ( mlt_destructor )mlt_producer_close, NULL );
- mlt_properties_set_data( properties, "transition", transition, 0, ( mlt_destructor )mlt_transition_close, NULL );
-
+
+ // Close our references
+ mlt_producer_close( track0 );
+ mlt_producer_close( track1 );
+ mlt_transition_close( transition );
+
// Return the tractor
return mlt_tractor_producer( tractor );
}
and we have a means to play multiple clips with a horribly obtrusive
watermark - just what the world needed, right? ;-)
+ Incidentally, the same thing could be achieved with the more trivial
+ watermark filter inserted between the producer and the consumer.
+
SECTION 3 - STRUCTURE AND DESIGN
--------------------------------
-Class Heirachy:
+Class Hierarchy:
- The mlt framework consists of an OO class heirachy which consists of the
+ The mlt framework consists of an OO class hierarchy which consists of the
following public classes and abstractions:
mlt_properties
too.
Steps 6 and 7 demonstrate that the properties object handles deserialisation
- from strings. The string value of "50" is set, the integer value of 50 is
+ from strings. The string value of "0" is set, the integer value of 0 is
retrieved.
Steps 8 and 9 demonstrate that the properties object handles serialisation
+------------------+------------------------------------+------------------+
|samples |The samples of the audio | |
+------------------+------------------------------------+------------------+
- |aspect_ratio |The aspect ratio of the image |double |
+ |aspect_ratio |The sample aspect ratio of the image|double |
+------------------+------------------------------------+------------------+
|test_image |Used to indicate no image available |0 or 1 |
+------------------+------------------------------------+------------------+
|rescale.interp |Use this scale method for test image|"string" |
+------------------+------------------------------------+------------------+
- While most of these are mainly self explainatory, the normalised_width and
- normalised_height values require a little explaination. These are required
+ While most of these are mainly self explanatory, the normalised_width and
+ normalised_height values require a little explanation. These are required
to ensure that effects are consistently handled as PAL or NTSC, regardless
of the consumers or producers width/height image request.
void *mlt_frame_pop_audio( mlt_frame this );
void mlt_frame_close( mlt_frame this );
-
+
mlt_service:
The service base class extends properties and allows 0 to m inputs and 0 to
A service does not define any properties when constructed. It should be
noted that producers, filters and transitions my be serialised (say, via the
- westley consumer), and care should be taken to distinguish between
+ xml consumer), and care should be taken to distinguish between
serialisable and transient properties. The convention used is to prefix
transient properties with an underscore.
void mlt_service_close( mlt_service this );
Typically, only direct descendents of services need invoke these methods and
- developers are encouraged to use those extensions when definining new
- services.
+ developers are encouraged to use those extensions when defining new services.
mlt_producer:
The public interface is defined by the following functions:
+ mlt_producer mlt_producer_new( );
int mlt_producer_init( mlt_producer this, void *child );
mlt_service mlt_producer_service( mlt_producer this );
mlt_properties mlt_producer_properties( mlt_producer this );
void mlt_producer_prepare_next( mlt_producer this );
void mlt_producer_close( mlt_producer this );
- For the sake of discussion here, we'll assume that someone wants to provide
- a new producer which random noise
-
- // Forward reference
- static int producer_get_frame( mlt_service service, mlt_frame_ptr frame, int index );
-
- mlt_producer producer_green( void *arg )
- {
- // Create a new producer
- mlt_producer this = mlt_producer_new( );
-
- // Check that we were allocated a new producer
- if ( this != NULL )
- {
- // Get the service
- mlt_service service = mlt_producer_service( this );
-
- // We need to override the get_frame method
- service->get_frame = producer_get_frame;
- }
-
- // Return this producer
- return this;
- }
-
- static int producer_get_frame( mlt_service service, mlt_frame_ptr frame, int index )
- {
- // Create a new frame
- *frame = mlt_frame_init( );
-
- // Specify the get_image
- mlt_frame_push_get_image( *frame, producer_get_image );
-
- // Specify the get_audio
- mlt_frame_push_audio( *frame, producer_get_audio );
-
- // Return that all was successful
- return 0;
- }
mlt_filter:
void mlt_playlist_close( mlt_playlist this );
mlt_tractor:
-
-
-mlt_factory
-
projects / mlt / blobdiff