Framework Documentation
-Copyright (C) 2004 Ushodaya Enterprises Limited
+Copyright (C) 2004-2009 Ushodaya Enterprises Limited
Author: Charles Yates <charles.yates@pandora.be>
-Last Revision: 2004-03-20
+Last Revision: 2005-05-08
MLT FRAMEWORK
This document is split roughly into 3 sections. The first section provides a
basic overview of MLT, the second section shows how it's used and the final
- section shows shows structure and design, with an emphasis on how the system
- is extended.
+ section shows structure and design, with an emphasis on how the system is
+ extended.
Target Audience:
opposed to the implementation details.
It is not required reading for the MLT client/server integration - please
- refer to valerie.txt and dvcp.txt for more details on this area.
+ refer to libmvsp.txt and mvsp.txt for more details on this area.
SECTION 1 - BASIC OVERVIEW
The defaults requested here are a special case - the NULL usage requests
that we use the default producers and consumers.
- The default producer is "fezzik". This producer matches file names to
+ The default producer is "loader". This producer matches file names to
locate a service to use and attaches 'normalising filters' (such as scalers,
deinterlacers, resamplers and field normalisers) to the loaded content -
these filters ensure that the consumer gets what it asks for.
- The default consumer is "sdl". The combination of fezzik and sdl will
+ The default consumer is "sdl". The combination of loader and sdl will
provide a media player.
In this example, we connect the producer and then start the consumer. We
Also note, you can override the defaults as follows:
- $ MLT_CONSUMER=westley ./hello file.avi
+ $ MLT_CONSUMER=xml ./hello file.avi
- This will create a westley xml document on stdout.
+ This will create a XML document on stdout.
- $ MLT_CONSUMER=westley MLT_PRODUCER=avformat ./hello file.avi
+ $ MLT_CONSUMER=xml MLT_PRODUCER=avformat ./hello file.avi
This will play the video using the avformat producer directly, thus it will
bypass the normalising functions.
+------------------+------------------------------------+------------------+
|MLT_NORMALISATION |The normalisation of the system |PAL or NTSC |
+------------------+------------------------------------+------------------+
- |MLT_PRODUCER |The default producer |"fezzik" or other |
+ |MLT_PRODUCER |The default producer |"loader" or other |
+------------------+------------------------------------+------------------+
|MLT_CONSUMER |The default consumer |"sdl" or other |
+------------------+------------------------------------+------------------+
+ |MLT_TEST_CARD |The default test card producer |any producer |
+ +------------------+------------------------------------+------------------+
These values are initialised from the environment variables of the same
name.
Instead of invoking mlt_factory_producer directly, we'll create a new
function called create_playlist. This function is responsible for creating
- the playlist, creating each producer, appending to the playlist and ensuring
- that all the producers are cleaned up when the playlist is destroyed. The
- last point is important - a close on the playlist won't explicitly close these
- producers. In this example, we use unique "data" properties with destructors
- to ensure closing.
+ the playlist, creating each producer and appending to the playlist.
mlt_producer create_playlist( int argc, char **argv )
{
int i = 0;
for ( i = 1; i < argc; i ++ )
{
- // Define the unique key
- char key[ 256 ];
-
// Create the producer
mlt_producer producer = mlt_factory_producer( NULL, argv[ i ] );
// Add it to the playlist
mlt_playlist_append( playlist, producer );
- // Create a unique key for this producer
- sprintf( key, "producer%d", i );
-
- // Now we need to ensure the producers are destroyed
- mlt_properties_set_data( properties, key, producer, 0, ( mlt_destructor )mlt_producer_close, NULL );
+ // Close the producer (see below)
+ mlt_producer_close( producer );
}
// Return the playlist as a producer
return mlt_playlist_producer( playlist );
}
+ Notice that we close the producer after the append. Actually, what we're
+ doing is closing our reference to it - the playlist creates its own reference
+ to the producer on append and insert, and it will close its reference
+ when the playlist is destroyed[*].
+
+ Note also that if you append multiple instances of the same producer, it
+ will create multiple references to it.
+
Now all we need do is to replace these lines in the main function:
// Create a normalised producer
and we have a means to play multiple clips.
+ [*] This reference functionality was introduced in mlt 0.1.2 - it is 100%
+ compatable with the early mechanism of registering the reference and
+ destructor with the properties of the playlist object.
+
Filters:
section, even multiple tracks have a single track output.
+Attached Filters:
+
+ All services can have attached filters.
+
+ Consider the following example:
+
+ // Create a producer
+ mlt_producer producer = mlt_factory_producer( NULL, clip );
+
+ // Get the service object of the producer
+ mlt_producer service = mlt_producer_service( producer );
+
+ // Create a filter
+ mlt_filter filter = mlt_factory_filter( "greyscale" );
+
+ // Create a playlist
+ mlt_playlist playlist = mlt_playlist_init( );
+
+ // Attach the filter to the producer
+ mlt_service_attach( producer, filter );
+
+ // Construct a playlist with various cuts from the producer
+ mlt_playlist_append_io( producer, 0, 99 );
+ mlt_playlist_append_io( producer, 450, 499 );
+ mlt_playlist_append_io( producer, 200, 399 );
+
+ // We can close the producer and filter now
+ mlt_producer_close( producer );
+ mlt_filter_close( filter );
+
+ When this is played out, the greyscale filter will be executed for each frame
+ in the playlist which comes from that producer.
+
+ Further, each cut can have their own filters attached which are executed after
+ the producer's filters. As an example:
+
+ // Create a new filter
+ filter = mlt_factory_filter( "invert", NULL );
+
+ // Get the second 'clip' in the playlist
+ producer = mlt_playlist_get_clip( 1 );
+
+ // Get the service object of the clip
+ service = mlt_producer_service( producer );
+
+ // Attach the filter
+ mlt_service_attach( producer, filter );
+
+ // Close the filter
+ mlt_filter_close( filter );
+
+ Even the playlist itself can have an attached filter:
+
+ // Create a new filter
+ filter = mlt_factory_filter( "watermark", "+Hello.txt" );
+
+ // Get the service object of the playlist
+ service = mlt_playlist_service( playlist );
+
+ // Attach the filter
+ mlt_service_attach( service, filter );
+
+ // Close the filter
+ mlt_filter_close( filter );
+
+ And, of course, the playlist, being a producer, can be cut up and placed on
+ another playlist, and filters can be attached to those cuts or on the new
+ playlist itself and so on ad nauseum.
+
+ The main advantage of attached filters is that they remain attached and don't
+ suffer from the maintenance problems associated with items being inserted and
+ displacing calculated in/out points - this being a major issue if you
+ exclusively use the connect or insert detached filters in a multitrack field
+ (described below).
+
+
+Introducing the Mix:
+
+ The mix is the simplest way to introduce transitions between adjacent clips
+ on a playlist.
+
+ Consider the following playlist:
+
+ +-+----------------------+----------------------------+-+
+ |X|A |B |X|
+ +-+----------------------+----------------------------+-+
+
+ Let's assume that the 'X' is a 'black clip' of 50 frames long.
+
+ When you play this out, you'll get a 50 frames of black, abrupt cut into
+ A, followed by an abrupt cut into B, and finally into black again.
+
+ The intention is to convert this playlist into something like:
+
+ +-+---------------------+-+------------------------+-+
+ |X|A |A|B |B|
+ |A| |B| |X|
+ +-+---------------------+-+------------------------+-+
+
+ Where the clips which refer to 2 clips represent a transition. Notice that
+ the representation of the second playlist is shorter than the first - this is
+ to be expected - a single transition of 50 frames between two clips will
+ reduce the playtime of the result by 50 frames.
+
+ This is done via the use of the mlt_playlist_mix method. So, assuming you get
+ a playlist as shown in the original diagram, to do the first mix, you could do
+ something like:
+
+ // Create a transition
+ mlt_transition transition = mlt_factor_transition( "luma", NULL );
+
+ // Mix the first and second clips for 50
+ mlt_playlist_mix( playlist, 0, 50, transition );
+
+ // Close the transition
+ mlt_transition_close( transition );
+
+ This would give you the first transition, subsequently, you would apply a similar
+ technique to mix clips 1 and 2. Note that this would create a new clip on the
+ playlist, so the next mix would be between 3 and 4.
+
+ As a general hint, to simplify the requirement to know the next clip index,
+ you might find the following simpler:
+
+ // Get the number of clips on the playlist
+ int i = mlt_playlist_count( );
+
+ // Iterate through them in reverse order
+ while ( i -- )
+ {
+ // Create a transition
+ mlt_transition transition = mlt_factor_transition( "luma", NULL );
+
+ // Mix the first and second clips for 50
+ mlt_playlist_mix( playlist, i, 50, transition );
+
+ // Close the transition
+ mlt_transition_close( transition );
+ }
+
+ There are other techniques, like using the mlt_playlist_join between the
+ current clip and the newly created one (you can determine if a new clip was
+ created by comparing the playlist length before and after the mix call).
+
+ Internally, the mlt_playlist_mix call generates a tractor and multitrack as
+ described below. Like the attached filters, the mix makes life very simple
+ when you're inserting items into the playlist.
+
+ Also note that it allows a simpler user interface - instead of enforcing the
+ use of a complex multitrack object, you can do many operations on a single
+ track. Thus, additional tracks can be used to introduce audio dubs, mixes
+ or composites which are independently positioned and aren't affected by
+ manipulations on other tracks. But hey, if you want a bombastic, confusing
+ and ultimately frustrating traditional NLE experience, that functionality
+ is provided too ;-).
+
+
+Practicalities and Optimisations:
+
+ In the previous two sections I've introduced some powerful functionality
+ designed to simplify MLT usage. However, a general issue comes into this -
+ what happens when you introduce a transition between two cuts from the same
+ bit of video footage?
+
+ Anyone who is familiar with video compression will be aware that seeking
+ isn't always without consequence from a performance point of view. So if
+ you happen to require two frames from the same clip for a transition, the
+ processing is going to be excessive and the result will undoubtedly be very
+ unpleasant, especially if you're rendering in realtime...
+
+ So how do we get round this?
+
+ Actually, it's very simple - you invoke mlt_producer_optimise on the top
+ level object after a modification and MLT will determine how to handle it.
+ Internally, it determines the maximum number of overlapping instances
+ throughout the object and creates clones and assigns clone indexes as
+ required.
+
+ In the mix example above, you can simply call:
+
+ // Optimise the playlist
+ mlt_producer_optimise( mlt_playlist_producer( playlist ) );
+
+ after the mix calls have be done. Note that this is automatically applied
+ to deserialised MLT XML.
+
+
Multiple Tracks and Transitions:
MLT's approach to multiple tracks is governed by two requirements:
+------------------------------+
The overlapping areas of track 0 and 1 would (presumably) have some kind of
- transition - without a transition, the frames from a1 and a2 would be shown
- during the areas of overlap.
+ transition - without a transition, the frames from b1 and b2 would be shown
+ during the areas of overlap (ie: by default, the higher numbered track takes
+ precedence over the lower numbered track).
MLT has a multitrack object, but it is not a producer in the sense that it
can be connected directly to a consumer and everything will work correctly.
+----------+
With a combination of the two, we can now connect multitracks to consumers.
- The first non-test card will be retrieved and passed on.
+ The last non-test card will be retrieved and passed on.
The tracks can be producers, playlists, or even other tractors.
tractor and the multitrack, but this involves a lot of connecting and
reconnecting left and right producers and consumers, and it seemed only fair
that we should be able to automate that process.
-
+
So in keeping with our agricultural theme, the concept of the 'field' was
born. We 'plant' filters and transitions in the field and the tractor pulls
the multitrack (think of a combine harvester :-)) over the field and
| +------+ | +-------------+ +-------+
+----------+
- In reality, we create a field first, and from that we obtain a multitrack
- and a tractor. We can then populate the multitrack, field and finally,
- connect the tractor to the consumer.
-
- The reasoning behind this is possibly flawed - it might have made more
- sense to produce the tractor and have it encapsulate the field and the
- multitrack as that is how it looks to a connected consumer:
+ So, we need to create the tractor first, and from that we obtain the
+ multitrack and field objects. We can populate these and finally
+ connect the tractor to a consumer.
+
+ In essence, this is how it looks to the consumer:
+-----------------------------------------------+
|tractor +--------------------------+ |
mlt_producer create_tracks( int argc, char **argv )
{
- // Create the field
- mlt_field field = mlt_field_init( );
+ // Create the tractor
+ mlt_tractor tractor = mlt_tractor_new( );
+
+ // Obtain the field
+ mlt_field field = mlt_tractor_field( tractor );
// Obtain the multitrack
- mlt_multitrack multitrack = mlt_field_multitrack( field );
-
- // Obtain the tractor
- mlt_tractor tractor = mlt_field_tractor( field );
+ mlt_multitrack multitrack = mlt_tractor_multitrack( tractor );
- // Obtain a composite transition
- mlt_transition transition = mlt_factory_transition( "composite", "10%,10%:15%x15%" );
+ // Create a composite transition
+ mlt_transition transition = mlt_factory_transition( "composite", "10%/10%:15%x15%" );
// Create track 0
mlt_producer track0 = create_playlist( argc, argv );
- // Create the watermark track - note we NEED fezzik for scaling here
- mlt_producer track1 = mlt_factory_producer( "fezzik", "pango" );
+ // Create the watermark track - note we NEED loader for scaling here
+ mlt_producer track1 = mlt_factory_producer( "loader", "pango" );
// Get the length of track0
mlt_position length = mlt_producer_get_playtime( track0 );
mlt_properties_set_position( properties, "in", 0 );
mlt_properties_set_position( properties, "out", length - 1 );
mlt_properties_set_position( properties, "length", length );
-
+ mlt_properties_set_int( properties, "a_track", 0 );
+ mlt_properties_set_int( properties, "b_track", 1 );
+
// Now set the properties on the transition
properties = mlt_transition_properties( transition );
mlt_properties_set_position( properties, "in", 0 );
// Now plant the transition
mlt_field_plant_transition( field, transition, 0, 1 );
-
- // Now set the properties on the tractor
- properties = mlt_tractor_properties( tractor );
- mlt_properties_set_data( properties, "multitrack", multitrack, 0, ( mlt_destructor )mlt_multitrack_close, NULL );
- mlt_properties_set_data( properties, "field", field, 0, ( mlt_destructor )mlt_field_close, NULL );
- mlt_properties_set_data( properties, "track0", track0, 0, ( mlt_destructor )mlt_producer_close, NULL );
- mlt_properties_set_data( properties, "track1", track1, 0, ( mlt_destructor )mlt_producer_close, NULL );
- mlt_properties_set_data( properties, "transition", transition, 0, ( mlt_destructor )mlt_transition_close, NULL );
-
+
+ // Close our references
+ mlt_producer_close( track0 );
+ mlt_producer_close( track1 );
+ mlt_transition_close( transition );
+
// Return the tractor
return mlt_tractor_producer( tractor );
}
and we have a means to play multiple clips with a horribly obtrusive
watermark - just what the world needed, right? ;-)
+ Incidentally, the same thing could be achieved with the more trivial
+ watermark filter inserted between the producer and the consumer.
+
SECTION 3 - STRUCTURE AND DESIGN
--------------------------------
A service does not define any properties when constructed. It should be
noted that producers, filters and transitions my be serialised (say, via the
- westley consumer), and care should be taken to distinguish between
+ xml consumer), and care should be taken to distinguish between
serialisable and transient properties. The convention used is to prefix
transient properties with an underscore.
projects / mlt / blobdiff