Document the imperative 1.9.0 functions.

[nageru-docs] / theme.rst

The theme
=========

**NOTE**: Nageru 1.9.0 made significant improvements to themes
and how scenes work. If you use an older version, you may want
to look at `the 1.8.6 documentation <https://nageru.sesse.net/doc-1.8.6/>`_;
themes written for older versions still work without modification in
1.9.0, but are not documented here, and you are advised to change
to use the new interfaces, as they are equally powerful and much simpler
to work with.

In Nageru, most of the business logic around how your stream
ends up looking is governed by the **theme**, much like how a
theme works on a blog or a CMS. Most importantly, the theme
governs the look and feel through the different scenes and
transitions between them, such that an event gets a consistent
visual language even if the operators differ. Instead of requiring the user
to modify Nageru's C++ core, themes are written in
`Lua <https://www.lua.org/>`_, a lightweight scripting language
made for embedding.

Themes contain a lot of logic, and writing one can seem a bit
daunting at first. However, most events will be happy just tweaking
one of the themes included with Nageru, and the operator (if different
from the visual designer) will not need to worry at all.

Nageru ships with two themes, a default full-featured two-camera
setup with side-by-side for e.g. conferences, and a minimal one
that is easier for new users to understand.


Introduction to scenes
----------------------

Anything that's shown on the stream, or on one of the preview displays,
is created by a **Movit chain**, instantiated by a Nageru **scene**.
`Movit <https://movit.sesse.net/>`_
is a library for high-quality, high-performance video filters,
and Nageru's themes can use a simplified version of Movit's API where
most of the low-level details are abstracted away.

Every frame, the theme chooses a **scene** and a set of parameters to it,
based on what it thinks the picture should look like. Every scene
consists of a set of *inputs* (which can be either live video streams
or static pictures) and then a set of operators or *effects* to combine
or modify each other. Movit compiles these down to a set of shaders
that run in high speed on the GPU; the theme doesn't see a pixel,
and thus, Lua's performance (even though good for its language class,
especially if you use `LuaJIT <http://luajit.org/>`_) will not matter
much.


Basic scenes
------------

The simplest possible scene takes only in an input and sends it on
to the display (the output of the last added node is always sent to
the screen, and in this case, that would be the input)::

  local scene = Scene.new(16, 9)  -- Aspect ratio.
  local input = scene:add_input()
  input:display(0)  -- First input card. Can be changed whenever you want.
  scene:finalize()

The live scene is always processed in full resolution (typically 720p)
and then scaled down for the GUI. Preview scenes are rendered in exactly
the resolution required, although of course, intermediate steps could be
bigger.


Setting parameters, and the get_scene entry point
-------------------------------------------------

Many effects support parameters that can vary per-frame. Imagine,
for instance, a theme where you want to supports two inputs and fading between
them. This means you will need a scene that produces two inputs and
produces a mix of them; Movit's *MixEffect* is exactly what you want here::

  local scene = EffectChain.new(16, 9)

  local input0 = scene:add_input()
  input0:display(0)
  local input1 = scene:add_input()
  input1:display(1)

  local mix_effect = scene:add_effect(MixEffect.new(), input0, input1)
  scene:finalize()

Every frame, Movit will call your **get_scene** function, which has
this signature:

  function get_scene(num, t, width, height, signals)

“width” and “height” are what you'd expect (the output resolution).
t contains the current stream time in seconds. “num” contains 0
for the live view, 1 for the preview view, and 2, 3, 4, … for each
of the individual stream previews. “signals“ contains a bit of
information about each input signal, like its current resolution
or frame rate.

get_scene in return should return a scene. However, before you do that,
you can set the parameters **strength_first** and **strength_second**
on it; for instance like this::

  function get_scene(num, t, width, height, signals)
    -- Assume num is 0 here; you will need to handle the other
    -- cases, too.
    local fade_progress = 0.0
    if t >= 1.0 and t >= 2.0:  -- Between 1 and 2 seconds; do the fade.
      fade_progress = t - 1.0
    elseif t >= 2.0:
      fade_progress = 1.0
    end

    mix_effect:set_float("strength_first", 1.0 - fade_progress)
    mix_effect:set_float("strength_second", fade_progress)
    return scene
  end

Note that in the case where fade_progress is 0.0 or 1.0 (you are just
showing one of the inputs), you are wasting GPU power by using the
fade scene; you should just return a simpler one-input scene instead.

The get_scene function is the backbone of every Nageru theme.
As we shall see, however, it may end up dealing with a fair bit
of complexity as the theme grows.


Scene variants and effect alternatives
--------------------------------------

Setting up and finalizing a scene is relatively fast, but it still
takes a measurable amount of CPU time, since it needs to create an OpenGL
shader and have it optimized by the driver; 50–100 ms is not uncommon.
Given that 60 fps means each frame is 16.7 ms, you cannot create new scenes in
get_scene; every scene you could be using must be created at program start,
when your theme is initialized.

For any nontrivial theme, there are a lot of possible scenes. Let's
return to the case of the MixEffect scene from the previous section.
Now let us assume that we could deal with signals that come in at
1080p instead of the native 720p. In this case, we will want a high-quality
scaler before mixing; *ResampleEffect* provides one::

  local scene = EffectChain.new(16, 9)

  local input0 = scene:add_input()
  input0:display(0)
  local input0_scaled = scene:add_optional_effect(ResampleEffect.new())  -- Implicitly uses input0.
  scene_or_input.resample_effect:set_int("width", 1280)  -- Could also be set in get_scene().
  scene_or_input.resample_effect:set_int("height", 720)
  input0_scaled:enable()  -- Enable or disable as needed.

  local input1 = scene:add_input()
  input1:display(1)
  local input1_scaled = ... -- Similarly here and the rest.

  -- The rest is unchanged.

Clearly, there are four options here; both inputs could be unscaled,
input0 could be scaled but not input1, input1 could be scaled but not input0,
or both could be scaled. That means four scenes. However, you don't need to
care about this; behind the scenes (no pun intended), Nageru will make all
four versions for you and choose the right one as you call enable() or
disable() on each effect.

Beyond simple on/off switches, an effect can have many *alternatives*,
by giving in an array of effects. For instance, it is usually pointless to use
the high-quality resampling provided by ResampleEffect for the on-screen
outputs; we can use *ResizeEffect* (a simpler scaling algorithm provided
directly by the GPU) that instead. The scaling is set up like this::

  local input0 = scene:add_input()
  input0:display(0)
  local input0_scaled = scene:add_effect({ResampleEffect.new(), ResizeEffect.new()})  -- Implicitly uses input0.
  scene_or_input.resample_effect:set_int("width", 1280)  -- Just like before.
  scene_or_input.resample_effect:set_int("height", 720)

  -- Pick one in get_scene() like this:
  input0_scaled:choose(ResizeEffect)

  -- Or by numerical index:
  input0_scaled:choose(1)  -- Chooses ResizeEffect

Note that add_effect returns its input for convenience.

Actually, add_optional_effect() is just a wrapper around add_effect() with
IdentityEffect as the other alternative, and disable() is a convenience version of
choose(IdentityEffect).

Actually, more versions are created than you'd immediately expect.
In particular, the output format for the live output and all previews are
different (Y'CbCr versus RGBA), which is also handled transparently for you.
Also, the inputs could be interlaced, or they could be images, or videos (see
:ref:`images` and :doc:`video`), creating many more options. Again, you
generally don't need to care about this; Movit will make sure each and every of
those generated scenes runs optimally on your GPU. However, if the
combinatorial explosion increases startup time beyond what you are comfortable
with, see :ref:`locking`.


Transitions
-----------

As we have seen, the theme is king when it determines what to show
on screen. However, ultimately, it wants to delegate that power
to the operator. The abstraction presented from the theme to the user
is in the form of **transitions**. Every frame, Nageru calls the
following Lua entry point::

  function get_transitions(t)

(t is again the stream time, but it is provided only for convenience;
not all themes would want to use it.) get_transitions must return an array of
(currently exactly) three strings, of which any can be blank. These three
strings are used as labels on one button each, and whenever the operator clicks
one of them, Nageru calls this function in the theme::

  function transition_clicked(num, t)

where “num” is 0, 1 or 2, and t is again the theme time.

It is expected that the theme will use this and its internal state
to provide the abstraction (or perhaps illusion) of transitions to
the user. For instance, a theme will know that the live stream is
currently showing input 0 and the preview stream is showing input 1.
In this case, it can use two of the buttons to offer “Cut“ or “Fade”
transitions to the user. If the user clicks the cut button, the theme
can simply switch input and previews, which will take immediate
effect on the next frame. However, if the user clicks the fade button,
state will need to be set up so that next time get_scene() runs,
it will return the scene with the MixEffect, until it determines
the transition is over and changes back to showing only one input
(presumably the new one).


.. _channels:

Channels
--------

In addition to the live and preview outputs, a theme can declare
as many individual **channels** as it wants. These are shown at the
bottom of the screen, and are intended for the operator to see
what they can put up on the preview (in a sense, a preview of the
preview).

The number of channels is determined by calling this function
once at the start of the program::

  Nageru.set_num_channels(2)

0 is allowed, but doesn't make a lot of sense. Live and preview comes in
addition to this.

Each channel will have a label on it; you set it by calling::

  Nageru.set_channel_name(2, "Side-by-side")

Here, channel is 2, 3, 4, etc.—by default, 0 is called “Live” and
1 is called “Preview”, and you probably don't need to change this.

Each channel has its own scene, starting from number 2 for the first one
(since 0 is live and 1 is preview). The simplest form is simply a direct copy
of an input, and most themes will include one such channel for each input.
(Below, we will see that there are more types of channels, however.)
Since the mapping between the channel UI element and inputs is so typical,
Nageru allows the theme to simply declare that a channel corresponds to
a given signal::

  Nageru.set_channel_signal(2, 0)
  Nageru.set_channel_signal(3, 1)

Here, channels 2 and 3 (the two first ones) correspond directly to inputs
0 and 1, respectively. The others don't, and return -1. The effect on the
UI is that the user can right-click on the channel and configure the input
that way; in fact, this is currently the only way to configure them.

Furthermore, channels can have a color, which is governed by Nageru calling
a function your theme::

  function channel_color(channel)
 
The theme should return a CSS color (e.g. “#ff0000”, or “cyan”) for each
channel when asked; it can vary from frame to frame. A typical use is to mark
the currently playing input as red, or the preview as green.

And finally, there are two entry points related to white balance::

  Nageru.set_supports_wb(2, true)
  function set_wb(channel, red, green, blue)

If the first function is called with a true value (at the start of the theme),
the channel will get a “Set WB” button next to it, which will activate a color
picker. When the user picks a color (ostensibly with a gray point), the second
function will be called (with the RGB values in linear light—not sRGB!),
and the theme can then use it to adjust the white balance for that channel.
The typical way to to this is to have a *WhiteBalanceEffect* on each input
and set its “neutral_color” parameter using the “set_vec3” function.


More complicated channels: Composites
-------------------------------------

Direct inputs are not the only kind of channels possible; again, any scene
can be output. The most common case is different kinds of **composites**,
typically showing side-by-side or something similar. The typical UI presented
to the user in this case is that you create a channel that consists of the
finished setup; you use ResampleEffect (or ResizeEffect for preview scenes),
*PaddingEffect* (to place the rectangles on the screen, one of them with a
transparent border) and then *OverlayEffect* (to get both on the screen at
the same time). Optionally, you can have a background image at the bottom,
and perhaps a logo at the top. This allows the operator to select a pre-made
composits, and then transition to and from it from a single camera view (or even
between different composites) as needed.

Transitions involving composites tend to be the most complicated parts of the theme
logic, but also make for the most distinct parts of your visual look.


.. _images:

Image inputs
------------

In addition to video inputs, Nageru supports static **image inputs**.
These work pretty much the same way as live video inputs. Recall that
you chose what input to display like this::

  input:display(0)

Image inputs are instead created by instantiating *ImageInput* and
displaying that::

  bg = ImageInput.new("bg.jpeg")  -- Once, at the start of the program.
  input:display(bg)  -- In get_scene().

All image types supported by FFmpeg are supported; if you give in a video,
only the first frame is used. The file is checked once every second,
so if you update the file on-disk, it will be available in Nageru without
a restart. (If the file contains an error, the update will be ignored.)
This allows you to e.g. have simple message overlays that you can change
without restarting Nageru.


.. _menus:

Theme menus
-----------

Complicated themes, especially those dealing with :doc:`HTML inputs <html>`,
may have needs for user control that go beyond those of transition buttons.
(An obvious example may be “reload the HTML file”.) For this reason,
themes can also set simple *theme menus*, which are always visible
no matter what inputs are chosen.

If a theme chooses to set a theme menu, it will be available on the
main menu bar under “Theme”; if not, it will be hidden. You can set
the menu at startup or at any other point, using a simple series of
labels and function references::

  function modify_aspect()
    -- Your code goes here.
  end

  function reload_html()
    html_input:reload()
  end

  ThemeMenu.set(
    { "Change &aspect", modify_aspect },
    { "&Reload overlay", reload_html }
  )

When the user chooses a menu entry, the given Lua function will
automatically be called. There are no arguments nor return values.

There currently is no support for checkboxes, submenus, input boxes
or the likes. However, do note that since the theme is written in unrestricted
Lua, so you can use e.g. `lua-http <https://github.com/daurnimator/lua-http>`_
to listen for external connections and accept more complicated inputs
from those.