minor typos

[mlt] / docs / westley.txt

WESTLEY
-------

Preamble:

        Westley is the MLT projects XML serialisation/deserialisation format -
        as such, it closely mirrors the internal structure of the MLT API. 


Introduction:

        A westley document is essentially a list of 'producers' - a producer is
        an mlt object which generates mlt frames (images and associated audio
        samples). 
        
        There are 3 types of producer:
        
        * Basic Producers - these are typically file or device oriented feeds; 
        * Playlists - these are arrangements of multiple producers;
        * Multitracks - these are the fx encapsulators.

        In the mlt model, producers are created and attached to 'consumers' -
        consumers are software playback components (such as SDL), or wrappers for
        hardware drivers (such as bluefish) or even the westley serialising
        consumer itself (the latter doesn't receive frames - it merely
        interrogates the connected producer for its configuration).

        Although westley was defined as a serialisation mechanism for instantiated 
        MLT components, this document will concentrate on the hand authoring of 
        westley documents.


Rules:

        As shall become apparent through the remainder of this document, the basic
        tenet of westley authoring is to organise the document in the following
        manner:

        1) create producer elements for each unique media clip in the project;
        2) create playlists for each track;
        3) create a multitrack and specify filters and transitions;
        4) adding global filters.

        While other uses of westley exist, the approach taken here is to maximise
        efficiency for complex projects. 


Basic Producers:

        The simplest westley document is:

        <westley>
          <producer id="producer0">
            <property name="resource">clip1.dv</property>
          </producer>
        </westley>

        The westley wrapping is of course superfluous here - loading this document
        with MLT is identical to loading the clip directly. 

        Of course, you can specify additional properties. For example, consider an
        MPEG file with multiple soundtracks - you could define a westley document to
        ensure that the second audio track is loaded:

        <westley>
          <producer id="producer0">
            <property name="resource">clip1.mpeg</property>
            <property name="audio_track">1</property>
          </producer>
        </westley>

        NB: This relies on the mpeg being handled by the avformat producer, rather
        than the mcmpeg one. See services.txt for more details.

        A more useful example comes with the pango producer for a text producer.

        TODO: pango example...

        Notes:

        1) It is better not to specify in/out points when defining basic producers
        as these can be specified in the playlists. The reasoning is that in/out
        restricts the amount of the clip available, and could lead to the same clip
        being loaded multiple times if you need different regions of the clip
        elsewhere;
        2) A westley can be specified as a resource, so westleys can naturally
        encapsulate other westleys.


Playlists:

        Playlists provide a 'collection' structure for producers. These can be used
        to define 'tracks' in the multitrack approach, or simple playlists for
        sequential, single track playout.

        As an example, the following defines two basic producers and a playlist with 3
        items:

        <westley>
          <producer id="producer0">
            <property name="resource">clip1.dv</property>
          </producer>
          <producer id="producer1">
            <property name="resource">clip2.dv</property>
          </producer>
          <playlist id="playlist0">
            <entry producer="producer0" in="0" out="2999"/>
            <entry producer="producer1" in="0" out="999"/>
            <entry producer="producer0" in="3000" out="6999"/>
          </playlist>
        </westley>

        Here we see how the playlist defines the in/out points of the basic
        producers.

        Notes:

        1) All in/out points are absolute frame positions relative to the producer
        being appended to the playlist;
        2) Westley documents are currently authored for a specific normalisation;
        3) The last 'producer' in the document is the default for play out;
        4) Playlists can reference the same producer multiple times. In/out regions
        do not need to be contiguous - duplication and skipping is acceptable.


Interlude - Introducing Multitracks:

        So far we've defined basic producers and playlists/tracks - the tractor is
        the element that allows us to arrange our tracks and specify filters and
        transitions. Similarly to a playlist, a tractor is a container.

        Note that MLT doesn't see a filter or a transition as a producer in the
        normal sense - filters and transitions are passive when it comes to seeking.
        Internally, seeks are carried out on the producers. This is an important
        point - MLT does not follow a traditional graph oriented model.

        Visualising an MLT tractor and it's interaction with the consumer will
        assist here:

        +----------------------------------------------+
        |tractor                                       |
        | +----------+    +-+    +-+    +-+    +-+     |
        | |multitrack|    |f|    |f|    |t|    |t|     |
        | | +------+ |    |i|    |i|    |r|    |r|     |
        | | |track0|-|--->|l|- ->|l|- ->|a|--->|a|\    |
        | | +------+ |    |t|    |t|    |n|    |n| \   |
        | |          |    |e|    |e|    |s|    |s|  \  |
        | | +------+ |    |r|    |r|    |i|    |i|   \ |    +--------+
        | | |track1|-|- ->|0|--->|1|--->|t|--->|t|-----|--->|consumer|
        | | +------+ |    | |    | |    |i|    |i|   / |    +--------+
        | |          |    | |    | |    |o|    |o|  /  |         ^
        | | +------+ |    | |    | |    |n|    |n| /   |         |
        | | |track2|-|- ->| |- ->| |--->|0|- ->|1|/    |         |
        | | +------+ |    | |    | |    | |    | |     |         |
        | +----------+    +-+    +-+    +-+    +-+     |         |
        +----------------------------------------------+         |
               ^                                                 |
               |                                                 |
        +-----------+                                            |
        |APPLICATION|--------------------------------------------+
        +-----------+

        Internally, all frames from all tracks pass through all the filters and
        transitions - these are told which tracks to deal and which regions of the
        tracks to work on. 

        Note that the application communicates with the producer - it can alter
        playback speed, position, or even which producer is connected to which
        consumer. 
        
        The consumer receives the first non-blank frame (see below). It has no say
        in the order in which gets them (the sdl consumer when used with inigo might 
        appear to be an exception - it isn't - it simply has a route back to the 
        application to allow the application to interpret key presses).

        Whether this is better or worse than a traditional graph approach is a moot
        point. The author of this document likes it anyway :-).


Tractors:

        To create a multitrack westley, we can use two playlists and introduce a
        tractor. For the purposes of demonstration, I'll add a filter here too:

        <westley>
          <producer id="producer0">
            <property name="resource">clip1.dv</property>
          </producer>
          <producer id="producer1">
            <property name="resource">clip2.dv</property>
          </producer>
          <playlist id="playlist0">
            <entry producer="producer0" in="0" out="2999"/>
            <blank length="1000"/>
            <entry producer="producer0" in="3000" out="6999"/>
          <playlist id="playlist1">
            <blank length="3000"/>
            <entry producer="producer1" in="0" out="999"/>
          </playlist>
          <tractor id="tractor0">
            <multitrack>
              <track producer="playlist0"/>
              <track producer="playlist1"/>
            </multitrack>
            <filter>
              <property name="track">0</property>
              <property name="mlt_service">greyscale</property>
            </filter>
          </tractor>
        </westley>
        
        Here we see that blank frames are inserted into the first playlist and a
        blank is provided at the beginning of the second - this can be visualised in
        the traditional timeline widget as follows:

        +-------+   +-------------+
        |a      |   |a            |
        +-------+---+-------------+
                |b  |
                +---+
        
        Adding the filter on the top track, gives us:

        +-------+   +-------------+
        |a      |   |a            |
        +-------+---+-------------+
        |greyscale                |
        --------+---+-------------+
                |b  |
                +---+
        
        Note that it's only applied to the visible parts of the top track.

        The requirement to apply a filter to the output, as opposed to a specific track 
        leads us to the final item in the Rules section above. As an example, let's
        assume we wish to watermark all output, then we could use the following:

        <westley>
          <producer id="producer0">
            <property name="resource">clip1.dv</property>
          </producer>
          <producer id="producer1">
            <property name="resource">clip2.dv</property>
          </producer>
          <playlist id="playlist0">
            <entry producer="producer0" in="0" out="2999"/>
            <blank length="1000"/>
            <entry producer="producer0" in="3000" out="6999"/>
          <playlist id="playlist1">
            <blank length="3000"/>
            <entry producer="producer1" in="0" out="999"/>
          </playlist>
          <tractor id="tractor0">
            <multitrack>
              <track producer="playlist0"/>
              <track producer="playlist1"/>
            </multitrack>
            <filter>
              <property name="track">0</property>
              <property name="mlt_service">greyscale</property>
            </filter>
          </tractor>
          <tractor id="tractor1">
            <multitrack>
              <track producer="tractor0"/>
            </multitrack>
            <filter>
              <property name="mlt_service">watermark</property>
              <property name="resource">watermark1.png</property>
            </filter>
          </tractor>
        </westley>
        
        Here we employ another tractor and we define a single track (being the
        tractor we previously defined) and apply a watermarking filter there.

        This is simply provided as an example - the watermarking functionality could
        be better handled at the playout stage itself (ie: as a filter automatically
        placed between all producers and the consumer).

        TODO: transition example


Flexibility:

        The information presented above is considered the MLT Westley "normal"
        form. This is the output generated by the westley consumer, for example,
        when used with inigo. It is the output generated when you use the
        "Westley to File" consumer in the demo script, which beginners will find
        most useful for learning to use westley XML. This section describes 
        alternative forms the westley producer accepts.

        First of all, the normal form is more of a linear format with producers
        and playlists defined prior to their usage in a multitrack. Westley
        also accepts a hierarchical format with producers as children of tracks
        or playlist entries and with playlists as children of tracks:

        <westley>
          <tractor>
            <multitrack>
              <track>
                <playlist>
                  <entry>
                    <producer>
                      <property name="resource">clip1.dv</property>
                    </producer>
                  </entry>
                </playlist>
              </track>
            </multitrack>
          </tractor>
        </westley>

        Obviously, this example is meant to demonstrate hierarchy and not effective
        use of playlist or multitrack!

        Secondly, as part of error handling, westley is forgiving if you fail to
        supply <tractor>, <track>, and <entry> where one can be understood. This
        affords an abbreviated syntax that is less verbose and perhaps less 
        intimidating for a human to read and understand. One can simplify the
        above example as:

        <westley>
          <multitrack>
            <playlist>
              <producer>
                <property name="resource">clip1.dv</property>
              </producer>
            </playlist>
          </multitrack>
        </westley>

        Yes, filters and transitions can be added to the above example after the
        closing multitrack tag (</multitrack>) because it is still enclosed within
        the westley body tags.

        If you specify in and out on a producer and it has been enclosed within
        an <entry> or <playlist>, then the edit points apply to the playlist
        entry and not to the producer itself. This facilitates re-use of media:

        <playlist>
          <producer id="clip1" in="25" out="78">
            <property name="resource">clip1.dv</property>
          </producer>
          <entry producer="clip1" in="119" out="347"/>
        </playlist>

        In the above example, the producer attribute of the entry element is 
        a reference to the preceding producer. All references must follow the 
        definition. The edit points supplied on the producer above will not affect
        the entry that references it below because westley knows the clip is a 
        playlist entry and optimises this situation. The advantage is that one
        does not need to determine every clip to be included ahead of time 
        and specify them outside the context of the mutlitrack timeline.

        This form of authoring will be easier for many to visualise as a non-linear 
        editor's timeline. Here is a more complex example:

        <westley>
          <multitrack>
            <playlist>
              <producer id="foo" in="100" out="149">
                <property name="resource">clip2.mpeg</property>
              </producer>
              <blank length="25"/>
              <entry producer="foo" in="10" out="59"/>
            </playlist>
            <playlist>
              <blank length="25"/>
              <producer id="bar" in="100" out="199">
                <property name="resource">clip3.mpeg</property>
              </producer>
              <entry out="99" producer="bar"/>
            </playlist>
          </multitrack>
          <filter mlt_service="greyscale" track="0"/>
          <transition mlt_service="luma" in="25" out="49" a_track="0" b_track="1"/>
          <transition mlt_service="luma" in="75" out="99" a_track="0" b_track="1">
            <property name="reverse" value="1"/>
          </transition>
        </westley>

        Did you notice something different in the last example? Properties can be 
        expressed using XML attributes on the element as well. However, only 
        non-service-specific properties are supported in this way. For example,
        "mlt_service" is available to any producer, filter, or transition. However,
        "resource" is actually service-specific. Notice the syntax of the last 
        property, on the last transition. Westley accepts property values using 
        the "value" attribute as well as using element text.

        We have seen a few different ways of expressing property values. There are
        a couple more for properties that can accept XML data. For example, the
        GDK pixbuf producer with librsvg can handle embedded SVG, and the Pango
        producer can handle embedded Pango markup. You can enclose the embedded XML
        using a CDATA section:

        <property name="resource"><![CDATA[ <svg>...</svg> ]]></property>

        Please ensure the opening CDATA tag immediately follows the opening
        property tag and that the section closing tag immediately precedes the
        closing property tag.

        However, westley can also accept inline embedded XML:

        <property name="resource">
          <svg>
          </svg>
        </property>

        Currently, there is no namespace handling so a conflict will occur only on 
        any embedded XML that contains an element named "property" because 
        westley collects embedded XML until it reaches a closing property tag.
        
        TODO: xml entities


Technique:

        If one finds the above hierarchical, abbreviated format intuitive,
        start with a simple template and fill and extend as needed:
                <westley>
                  <multitrack>
                    <playlist>
                    </playlist>
                    ...add a playlist for each track...
                  </multitrack>
                  ...add filters and transitions...
                </westley>
                
        By using a playlist for each track, it is easier to iteratively add new 
        clips and blank regions as you develop the project.     You will not have to
        use <track> or later add <playlist> when necessary.
        
        A more advanced template that allows sequencing multitracks is:
                <playlist>
                  <entry>
                    <multitrack>
                      <playlist>
                      </playlist>
                      ...add a playlist for each track...
                    </multitrack>
                    ...add filters and transitions...
                  </entry>
                  
                  <entry>
                    <multitrack>
                      <playlist>
                      </playlist>
                      ...add a playlist for each track...
                    </multitrack>
                    ...add filters and transitions...
                  </entry>
                </playlist>

        If you want to have a silent, black background for audio and video fades,
        then make the last track simply <producer mlt_service="colour"/>. Then,
        use composite and volume key-framable properties. 
        TODO: to be continued
        
        TODO: considerations with mixing multiple audio layers