--- /dev/null
+The theme
+=========
+
+In Nageru, most of the business logic around how your stream
+ends up looking is governed by the **theme**, much like how a
+theme works on a blog or a CMS. Most importantly, the theme
+governs the look and feel through the different scenes and
+transitions between them, such that an event gets a consistent
+visual language even if the operators differ. Instead of requiring the user
+to modify Nageru's C++ core, themes are written in
+`Lua <https://www.lua.org/>`_, a lightweight scripting language
+made for embedding.
+
+Themes contain a lot of logic, and writing one can seem a bit
+daunting at first. However, most events will be happy just tweaking
+one of the themes included with Nageru, and the operator (if different
+from the visual designer) will not need to worry at all.
+
+Nageru ships with two themes, a default full-featured two-camera
+setup with side-by-side for e.g. conferences, and a minimal one
+that is easier for new users to understand.
+
+
+Introduction to chains
+----------------------
+
+Anything that's shown on the stream, or on one of the preview displays,
+is created by a **Movit chain**. `Movit <https://movit.sesse.net/>`_
+is a library for high-quality, high-performance video filters,
+and Nageru's themes can use a simplified version of Movit's API where
+most of the low-level details are abstracted away.
+
+Every frame, the theme choses a chain and a set of parameters to it,
+based on what it thinks the picture should look like. Every chain
+consists of a set of *inputs* (which can be either live video streams
+or static pictures) and then a set of operators or *effects* to combine
+or modify each other. Movit compiles these down to a set of shaders
+that run in high speed on the GPU; the theme doesn't see a pixel,
+and thus, Lua's performance (even though good for its language class,
+especially if you use `LuaJIT <http://luajit.org/>`_) will not matter
+much.
+
+
+High- and low-quality chains
+----------------------------
+
+The simplest possible chain takes only in an input and sends it on
+to the display (the output of the last added node is always sent to
+the screen, and in this case, that would be the input)::
+
+ local chain = EffectChain.new(16, 9) -- Aspect ratio.
+ local input = chain:add_live_input(false, false) -- No bounce override, no deinterlacing.
+ input:connect_signal(0) -- First input card. Can be changed whenever you want.
+ chain:finalize(hq)
+
+Note the “hq” parameter. Every chain needs to be able to run in two
+situations: Both for the stream output (the ”live” pane) and for the preview
+displays (the big “preview” pane and the smaller ones). The details have
+to do with Nageru internals (high-quality chains need to have an additional
+Y'CbCr output), but the distinction is also useful for themes. In particular,
+some operations, like scaling, can be done in various quality levels,
+and for a low-resolution preview display, you don't need maximum quality.
+Thus, in a preview chain (hq=false), you can safely take shortcuts.
+
+The live chain is always processed in full resolution (typically 720p)
+and then scaled down for the GUI. Preview chains are rendered in exactly
+the resolution required, although of course, intermediate steps could be
+bigger.
+
+Setting parameters, and the get_chain entry point
+-------------------------------------------------
+
+Many effects support parameters that can vary per-frame. Imagine,
+for instance, a theme where you want to supports two inputs and fading between
+them. This means you will need a chain that produces two inputs and
+produces a mix of them; Movit's *MixEffect* is exactly what you want here::
+
+ local chain = EffectChain.new(16, 9)
+
+ local input0 = chain:add_live_input(false, false)
+ input0:connect_signal(0)
+ local input1 = chain:add_live_input(false, false)
+ input1:connect_signal(1)
+
+ local mix_effect = chain:add_effect(MixEffect.new(), input0, input1)
+ chain:finalize(hq)
+
+Every frame, Movit will call your **get_chain** function, which has
+this signature:
+
+ function get_chain(num, t, width, height, signals)
+
+“width” and “height” are what you'd expect (the output resolution).
+t contains the current stream time in seconds. “num” contains 0
+for the live view, 1 for the preview view, and 2, 3, 4, … for each
+of the individual stream previews. “signals“ contains a bit of
+information about each input signal, like its current resolution
+or frame rate.
+
+get_chain is in turn responsible for returning two values:
+
+ * The first return value is a Movit chain, as described in these
+ sections. For the live stream (num=0), you should return a high-quality
+ chain; for all others, you should return a low-quality chain.
+ * The second parameter is an *closure* that will be called just before
+ the chain is to be rendered. (The same chain could be used in
+ multiple OpenGL contexts at the same time, so you can't just set the values
+ immediately before returning. If you set them in the closure,
+ Nageru and Movit will deal with all the required threading for you.)
+
+In the returned closure, you can set the parameters **strength_first**
+and **strength_second**; for instance like this::
+
+ function get_chain(num, t, width, height, signals)
+ -- Assume num is 0 here; you will need to handle the other
+ -- cases, too.
+ prepare = function()
+ input0:connect_signal(0)
+ input1:connect_signal(1)
+
+ local fade_progress = 0.0
+ if t >= 1.0 and t >= 2.0: -- Between 1 and 2 seconds; do the fade.
+ fade_progress = t - 1.0
+ elseif t >= 2.0:
+ fade_progress = 1.0
+ end
+
+ mix_effect:set_float("strength_first", 1.0 - fade_progress)
+ mix_effect:set_float("strength_second", fade_progress)
+ end
+ return chain, prepare
+ end
+
+Note that in the case where fade_progress is 0.0 or 1.0 (you are just
+showing one of the inputs), you are wasting GPU power by using the
+fade chain; you should just return a simpler one-input chain instead.
+
+The get_chain function is the backbone of every Nageru theme.
+As we shall see, however, it may end up dealing with a fair bit
+of complexity as the theme grows.
+
+
+Chain precalculation
+--------------------
+
+Setting up and finalizing a chain is relatively fast, but it still
+takes a measurable amount of CPU time, since it needs to create an OpenGL
+shader and have it optimized by the driver; 50–100 ms is not uncommon.
+Given that 60 fps means each frame is 16.7 ms, you cannot create new chains in
+get_chain; every chain you could be using must be created at program start,
+when your theme is initialized.
+
+For any nontrivial theme, there are a lot of possible chains. Let's
+return to the case of the MixEffect chain from the previous section.
+Now let us assume that we could deal with signals that come in at
+1080p instead of the native 720p. In this case, we will want a high-quality
+scaler before mixing; *ResampleEffect* provides one::
+
+ local chain = EffectChain.new(16, 9)
+
+ local input0 = chain:add_live_input(false, false)
+ input0:connect_signal(0)
+ local input0_scaled = chain:add_effect(ResampleEffect.new()) -- Implicitly uses input0.
+ chain_or_input.resample_effect:set_int("width", 1280) -- Would normally be set in the prepare function.
+ chain_or_input.resample_effect:set_int("height", 720)
+
+ local input1 = chain:add_live_input(false, false)
+ input1:connect_signal(1)
+
+ -- The rest is unchanged.
+
+Clearly, there are four options here; both inputs could be unscaled,
+input0 could be scaled but not input1, input1 could be scaled but not input0,
+or both could be scaled. That means four chains.
+
+Now remember that we need to create all your chains both in high-
+and low-quality versions. In particular, this determines the “hq”
+parameter to finalize(), but in our case, we would want to replace
+ResampleEffect by *ResizeEffect* (a simpler scaling algorithm provided
+directly by the GPU) for the low-quality versions. This makes for
+eight chains.
+
+Now also consider that we would want to deal with *interlaced*
+inputs. (You can check if you get an interlaced input on the Nth
+input by calling “signals:get_deinterlaced(n)” from get_chain.)
+This further quadruples the number of chains you'd need to write,
+and this isn't even including that you'd want the static chains.
+It is obvious that this should not be done by hand. The default
+included theme contains a handy Lua shortcut called
+**make_cartesian_product** where you can declare all the dimensions
+you would want to specialize your chain over, and have a callback
+function called for each possible combination. Movit will make sure
+each and every of those generated chains runs optimally on your GPU.
+
+
+Transitions
+-----------
+
+As we have seen, the theme is king when it determines what to show
+on screen. However, ultimately, it wants to delegate that power
+to the operator. The abstraction presented from the theme to the user
+is in the form of **transitions**. Every frame, Nageru calls the
+following Lua entry point::
+
+ function get_transitions(t)
+
+(t is again the stream time, but it is provided only for convenience;
+not all themes would want to use it.) get_transitions must return an array of
+(currently exactly) three strings, of which any can be blank. These three
+strings are used as labels on one button each, and whenever the operator clicks
+one of them, Nageru calls this function in the theme::
+
+ function transition_clicked(num, t)
+
+where “num” is 0, 1 or 2, and t is again the theme time.
+
+It is expected that the theme will use this and its internal state
+to provide the abstraction (or perhaps illusion) of transitions to
+the user. For instance, a theme will know that the live stream is
+currently showing input 0 and the preview stream is showing input 1.
+In this case, it can use two of the buttons to offer “Cut“ or “Fade”
+transitions to the user. If the user clicks the cut button, the theme
+can simply switch input and previews, which will take immediate
+effect on the next frame. However, if the user clicks the fade button,
+state will need to be set up so that next time get_chain() runs,
+it will return the chain with the MixEffect, until it determines
+the transition is over and changes back to showing only one input
+(presumably the new one).
+
+
+Channels
+--------
+
+In addition to the live and preview outputs, a theme can declare
+as many individual **channels** as it wants. These are shown at the
+bottom of the screen, and are intended for the operator to see
+what they can put up on the preview (in a sense, a preview of the
+preview).
+
+The number of channels is determined by calling this function
+once at the start of the program::
+
+ function num_channels()
+
+It should simply return the number of channels (0 is allowed,
+but doesn't make a lot of sense). Live and preview comes in addition to this.
+
+Each channel will have a label on it; Nageru asks the theme
+by calling this function::
+
+ function channel_name(channel)
+
+Here, channel is 2, 3, 4, etc.—0 is always called “Live” and
+1 is always called “Preview”.
+
+Each channel has its own chain, starting from number 2 for the first one
+(since 0 is live and 1 is preview). The simplest form is simply a direct copy
+of an input, and most themes will include one such channel for each input.
+(Below, we will see that there are more types of channels, however.)
+Since the mapping between the channel UI element and inputs is so typical,
+Nageru allows the theme to simply declare that a channel corresponds to
+a given signal, by asking it::
+
+ function channel_signal(channel)
+ if channel == 2 then
+ return 0
+ elseif channel == 3 then
+ return 1
+ else
+ return -1
+ end
+ end
+
+Here, channels 2 and 3 (the two first ones) correspond directly to inputs
+0 and 1, respectively. The others don't, and return -1. The effect on the
+UI is that the user can right-click on the channel and configure the input
+that way; in fact, this is currently the only way to configure them.
+
+Furthermore, channels can have a color::
+
+ function channel_color(channel)
+
+The theme should return a CSS color (e.g. “#ff0000”, or “cyan”) for each
+channel when asked; it can vary from frame to frame. A typical use is to mark
+the currently playing input as red, or the preview as green.
+
+And finally, there are two entry points related to white balance::
+
+ function supports_set_wb(channel)
+ function set_wb(channel, red, green, blue)
+
+If the first function returns true (called once, at the start of the program),
+the channel will get a “Set WB” button next to it, which will activate a color
+picker. When the user picks a color (ostensibly with a gray point), the second
+function will be called (with the RGB values in linear light—not sRGB!),
+and the theme can then use it to adjust the white balance for that channel.
+The typical way to to this is to have a *WhiteBalanceEffect* on each input
+and set its “neutral_color” parameter using the “set_vec3” function.
+
+
+More complicated channels: Scenes
+---------------------------------
+
+Direct inputs are not the only kind of channels possible; again, any chain
+can be output. The most common case is different kinds of **scenes**,
+typically showing side-by-side or something similar. The typical UI presented
+to the user in this case is that you create a channel that consists of the
+finished setup; you use ResampleEffect (or ResizeEffect for low-quality chains),
+*PaddingEffect* (to place the rectangles on the screen, one of them with a
+transparent border) and then *OverlayEffect* (to get both on the screen at
+the same time). Optionally, you can have a background image at the bottom,
+and perhaps a logo at the top. This allows the operator to select a pre-made
+scene, and then transition to and from it from a single camera view (or even
+between different scenes) as needed.
+
+Transitions involving scenes tend to be the most complicated parts of the theme
+logic, but also make for the most distinct parts of your visual look.
+
+
+Image inputs
+------------
+
+In addition to video inputs, Nageru supports static **image inputs**.
+These work pretty much the same way as live video inputs; however,
+they need to be instantiated in a different way. Recall that live inputs
+were created like this::
+
+ input = chain:add_live_input(false, deint)
+
+Image inputs are instead created by instantiating **ImageInput** and
+adding them manually to the chain::
+
+ input = chain:add_effect(ImageInput.new("bg.jpeg"))
+
+Note that add_effect returns its input for convenience.
+
+All image types supported by FFmpeg are supported; if you give in a video,
+only the first frame is used. The file is checked once every second,
+so if you update the file on-disk, it will be available in Nageru without
+a restart. (If the file contains an error, the update will be ignored.)
+This allows you to e.g. have simple message overlays that you can change
+without restarting Nageru.
projects / nageru-docs / commitdiff