The theme
=========
+**NOTE**: Nageru 1.9.0 made significant improvements to themes
+and how scenes work. If you use an older version, you may want
+to look at `the 1.8.6 documentation <https://nageru.sesse.net/doc-1.8.6/>`_;
+themes written for older versions still work without modification in
+1.9.0, but are not documented here, and you are advised to change
+to use the new interfaces, as they are equally powerful and much simpler
+to work with.
+
In Nageru, most of the business logic around how your stream
ends up looking is governed by the **theme**, much like how a
theme works on a blog or a CMS. Most importantly, the theme
that is easier for new users to understand.
-Introduction to chains
+Introduction to scenes
----------------------
Anything that's shown on the stream, or on one of the preview displays,
-is created by a **Movit chain**. `Movit <https://movit.sesse.net/>`_
+is created by a **Movit chain**, instantiated by a Nageru **scene**.
+`Movit <https://movit.sesse.net/>`_
is a library for high-quality, high-performance video filters,
and Nageru's themes can use a simplified version of Movit's API where
most of the low-level details are abstracted away.
-Every frame, the theme chooses a chain and a set of parameters to it,
-based on what it thinks the picture should look like. Every chain
+Every frame, the theme chooses a **scene** and a set of parameters to it,
+based on what it thinks the picture should look like. Every scene
consists of a set of *inputs* (which can be either live video streams
or static pictures) and then a set of operators or *effects* to combine
or modify each other. Movit compiles these down to a set of shaders
much.
-High- and low-quality chains
-----------------------------
+Basic scenes
+------------
-The simplest possible chain takes only in an input and sends it on
+The simplest possible scene takes only in an input and sends it on
to the display (the output of the last added node is always sent to
the screen, and in this case, that would be the input)::
- local chain = EffectChain.new(16, 9) -- Aspect ratio.
- local input = chain:add_live_input(false, false) -- No bounce override, no deinterlacing.
- input:connect_signal(0) -- First input card. Can be changed whenever you want.
- chain:finalize(hq)
-
-Note the “hq” parameter. Every chain needs to be able to run in two
-situations: Both for the stream output (the ”live” pane) and for the preview
-displays (the big “preview” pane and the :ref:`channels <channels>`). The details have
-to do with Nageru internals (high-quality chains need to have an additional
-Y'CbCr output), but the distinction is also useful for themes. In particular,
-some operations, like scaling, can be done in various quality levels,
-and for a low-resolution preview display, you don't need maximum quality.
-Thus, in a preview chain (hq=false), you can safely take shortcuts.
-
-The live chain is always processed in full resolution (typically 720p)
-and then scaled down for the GUI. Preview chains are rendered in exactly
+ local scene = Scene.new(16, 9) -- Aspect ratio.
+ local input = scene:add_input()
+ input:display(0) -- First input card. Can be changed whenever you want.
+ scene:finalize()
+
+The live scene is always processed in full resolution (typically 720p)
+and then scaled down for the GUI. Preview scenes are rendered in exactly
the resolution required, although of course, intermediate steps could be
bigger.
-Setting parameters, and the get_chain entry point
+Setting parameters, and the get_scene entry point
-------------------------------------------------
Many effects support parameters that can vary per-frame. Imagine,
for instance, a theme where you want to supports two inputs and fading between
-them. This means you will need a chain that produces two inputs and
+them. This means you will need a scene that produces two inputs and
produces a mix of them; Movit's *MixEffect* is exactly what you want here::
- local chain = EffectChain.new(16, 9)
+ local scene = EffectChain.new(16, 9)
- local input0 = chain:add_live_input(false, false)
- input0:connect_signal(0)
- local input1 = chain:add_live_input(false, false)
- input1:connect_signal(1)
+ local input0 = scene:add_input()
+ input0:display(0)
+ local input1 = scene:add_input()
+ input1:display(1)
- local mix_effect = chain:add_effect(MixEffect.new(), input0, input1)
- chain:finalize(hq)
+ -- Note that add_effect returns its input for convenience.
+ local mix_effect = scene:add_effect(MixEffect.new(), input0, input1)
+ scene:finalize()
-Every frame, Movit will call your **get_chain** function, which has
+Every frame, Movit will call your **get_scene** function, which has
this signature:
- function get_chain(num, t, width, height, signals)
+ function get_scene(num, t, width, height, signals)
“width” and “height” are what you'd expect (the output resolution).
t contains the current stream time in seconds. “num” contains 0
for the live view, 1 for the preview view, and 2, 3, 4, … for each
-of the individual stream previews. “signals“ contains a bit of
-information about each input signal, like its current resolution
-or frame rate.
-
-get_chain is in turn responsible for returning two values:
+of the individual stream previews. “signals” contains a bit of
+information about each input signal (see :ref:`signal-info`).
- * The first return value is a Movit chain, as described in these
- sections. For the live stream (num=0), you should return a high-quality
- chain; for all others, you should return a low-quality chain.
- * The second parameter is an *closure* that will be called just before
- the chain is to be rendered. (The same chain could be used in
- multiple OpenGL contexts at the same time, so you can't just set the values
- immediately before returning. If you set them in the closure,
- Nageru and Movit will deal with all the required threading for you.)
+get_scene in return should return a scene. However, before you do that,
+you can set the parameters **strength_first** and **strength_second**
+on it; for instance like this::
-In the returned closure, you can set the parameters **strength_first**
-and **strength_second**; for instance like this::
-
- function get_chain(num, t, width, height, signals)
+ function get_scene(num, t, width, height, signals)
-- Assume num is 0 here; you will need to handle the other
-- cases, too.
- prepare = function()
- input0:connect_signal(0)
- input1:connect_signal(1)
-
- local fade_progress = 0.0
- if t >= 1.0 and t >= 2.0: -- Between 1 and 2 seconds; do the fade.
- fade_progress = t - 1.0
- elseif t >= 2.0:
- fade_progress = 1.0
- end
-
- mix_effect:set_float("strength_first", 1.0 - fade_progress)
- mix_effect:set_float("strength_second", fade_progress)
+ local fade_progress = 0.0
+ if t >= 1.0 and t >= 2.0: -- Between 1 and 2 seconds; do the fade.
+ fade_progress = t - 1.0
+ elseif t >= 2.0:
+ fade_progress = 1.0
end
- return chain, prepare
+
+ mix_effect:set_float("strength_first", 1.0 - fade_progress)
+ mix_effect:set_float("strength_second", fade_progress)
+ return scene
end
Note that in the case where fade_progress is 0.0 or 1.0 (you are just
showing one of the inputs), you are wasting GPU power by using the
-fade chain; you should just return a simpler one-input chain instead.
+fade scene; you should just return a simpler one-input scene instead.
-The get_chain function is the backbone of every Nageru theme.
+The get_scene function is the backbone of every Nageru theme.
As we shall see, however, it may end up dealing with a fair bit
of complexity as the theme grows.
-Chain precalculation
---------------------
+Scene variants and effect alternatives
+--------------------------------------
-Setting up and finalizing a chain is relatively fast, but it still
+Setting up and finalizing a scene is relatively fast, but it still
takes a measurable amount of CPU time, since it needs to create an OpenGL
shader and have it optimized by the driver; 50–100 ms is not uncommon.
-Given that 60 fps means each frame is 16.7 ms, you cannot create new chains in
-get_chain; every chain you could be using must be created at program start,
+Given that 60 fps means each frame is 16.7 ms, you cannot create new scenes in
+get_scene; every scene you could be using must be created at program start,
when your theme is initialized.
-For any nontrivial theme, there are a lot of possible chains. Let's
-return to the case of the MixEffect chain from the previous section.
+For any nontrivial theme, there are a lot of possible scenes. Let's
+return to the case of the MixEffect scene from the previous section.
Now let us assume that we could deal with signals that come in at
1080p instead of the native 720p. In this case, we will want a high-quality
scaler before mixing; *ResampleEffect* provides one::
- local chain = EffectChain.new(16, 9)
+ local scene = EffectChain.new(16, 9)
- local input0 = chain:add_live_input(false, false)
- input0:connect_signal(0)
- local input0_scaled = chain:add_effect(ResampleEffect.new()) -- Implicitly uses input0.
- chain_or_input.resample_effect:set_int("width", 1280) -- Would normally be set in the prepare function.
- chain_or_input.resample_effect:set_int("height", 720)
+ local input0 = scene:add_input()
+ input0:display(0)
+ local input0_scaled = scene:add_optional_effect(ResampleEffect.new()) -- Implicitly uses input0.
+ input0_scaled:set_int("width", 1280) -- Could also be set in get_scene().
+ input0_scaled:set_int("height", 720)
+ input0_scaled:enable() -- Enable or disable as needed.
- local input1 = chain:add_live_input(false, false)
- input1:connect_signal(1)
+ local input1 = scene:add_input()
+ input1:display(1)
+ local input1_scaled = ... -- Similarly here and the rest.
+
+ input1_scaled:enable_if(some_variable) -- Convenience form for enable() or disable() depending on some_variable.
-- The rest is unchanged.
Clearly, there are four options here; both inputs could be unscaled,
input0 could be scaled but not input1, input1 could be scaled but not input0,
-or both could be scaled. That means four chains.
-
-Now remember that we need to create all your chains both in high-
-and low-quality versions. In particular, this determines the “hq”
-parameter to finalize(), but in our case, we would want to replace
-ResampleEffect by *ResizeEffect* (a simpler scaling algorithm provided
-directly by the GPU) for the low-quality versions. This makes for
-eight chains.
-
-Now also consider that we would want to deal with *interlaced*
-inputs. (You can check if you get an interlaced input on the Nth
-input by calling “signals:get_deinterlaced(n)” from get_chain.)
-This further quadruples the number of chains you'd need to write,
-and this isn't even including that you'd want the static chains.
-It is obvious that this should not be done by hand. The default
-included theme contains a handy Lua shortcut called
-**make_cartesian_product** where you can declare all the dimensions
-you would want to specialize your chain over, and have a callback
-function called for each possible combination. Movit will make sure
-each and every of those generated chains runs optimally on your GPU.
+or both could be scaled. That means four scenes. However, you don't need to
+care about this; behind the scenes (no pun intended), Nageru will make all
+four versions for you and choose the right one as you call enable() or
+disable() on each effect.
+
+Beyond simple on/off switches, an effect can have many *alternatives*,
+by giving in an array of effects. For instance, it is usually pointless to use
+the high-quality resampling provided by ResampleEffect for the on-screen
+outputs; we can use *ResizeEffect* (a simpler scaling algorithm provided
+directly by the GPU) that instead. The scaling is set up like this::
+
+ local input0 = scene:add_input()
+ input0:display(0)
+ local input0_scaled = scene:add_effect({ResampleEffect.new(), ResizeEffect.new()}) -- Implicitly uses input0.
+ input0_scaled:set_int("width", 1280) -- Just like before.
+ input0_scaled:set_int("height", 720)
+
+ -- Pick one in get_scene() like this:
+ input0_scaled:choose(ResizeEffect)
+
+ -- Or by numerical index:
+ input0_scaled:choose(1) -- Chooses ResizeEffect
+
+Actually, add_optional_effect() is just a wrapper around add_effect() with
+IdentityEffect as the other alternative, and disable() is a convenience version of
+choose(IdentityEffect).
+
+All alternatives must
+have the same amount of inputs, with an exception for IdentityEffect, which can
+coexist with an effect requiring any amount of inputs (if selected, the IdentityEffect
+just passes its first input unchanged). Similarly, if you set a parameter with
+set_int() or similar, it must be valid for all alternatives (again excepting
+IdentityEffect); if there is one that can only be used on a certain alternative,
+you must set it directly on the effect::
+
+ local resample_effect = ResampleEffect.new()
+ resample_effect:set_float("zoom_x", 1.0001) -- Not valid for ResizeEffect.
+
+ local input0_scaled = scene:add_effect({resample_effect, ResizeEffect.new()})
+ input0_scaled:set_int("width", 1280) -- Set on both alternatives.
+ input0_scaled:set_int("height", 720)
+
+ -- This is also possible, as choose() returns the chosen effect:
+ input0_scaled:choose(ResampleEffect):set_float("zoom_y", 1.0001)
+
+Actually, more versions are created than you'd immediately expect.
+In particular, the output format for the live output and all previews are
+different (Y'CbCr versus RGBA), which is also handled transparently for you.
+Also, the inputs could be interlaced, or they could be images, or videos (see
+:ref:`images` and :doc:`video`), creating many more options. Again, you
+generally don't need to care about this; Movit will make sure each and every of
+those generated scenes runs optimally on your GPU. However, if the
+combinatorial explosion increases startup time beyond what you are comfortable
+with, see :ref:`locking`.
Transitions
transitions to the user. If the user clicks the cut button, the theme
can simply switch input and previews, which will take immediate
effect on the next frame. However, if the user clicks the fade button,
-state will need to be set up so that next time get_chain() runs,
-it will return the chain with the MixEffect, until it determines
+state will need to be set up so that next time get_scene() runs,
+it will return the scene with the MixEffect, until it determines
the transition is over and changes back to showing only one input
(presumably the new one).
The number of channels is determined by calling this function
once at the start of the program::
- function num_channels()
+ Nageru.set_num_channels(2)
-It should simply return the number of channels (0 is allowed,
-but doesn't make a lot of sense). Live and preview comes in addition to this.
+0 is allowed, but doesn't make a lot of sense. Live and preview comes in
+addition to this.
-Each channel will have a label on it; Nageru asks the theme
-by calling this function::
+Each channel will have a label on it; you set it by calling::
- function channel_name(channel)
+ Nageru.set_channel_name(2, "Side-by-side")
-Here, channel is 2, 3, 4, etc.—0 is always called “Live” and
-1 is always called “Preview”.
+Here, channel is 2, 3, 4, etc.—by default, 0 is called “Live” and
+1 is called “Preview”, and you probably don't need to change this.
-Each channel has its own chain, starting from number 2 for the first one
+Each channel has its own scene, starting from number 2 for the first one
(since 0 is live and 1 is preview). The simplest form is simply a direct copy
of an input, and most themes will include one such channel for each input.
(Below, we will see that there are more types of channels, however.)
Since the mapping between the channel UI element and inputs is so typical,
Nageru allows the theme to simply declare that a channel corresponds to
-a given signal, by asking it::
+a given signal::
- function channel_signal(channel)
- if channel == 2 then
- return 0
- elseif channel == 3 then
- return 1
- else
- return -1
- end
- end
+ Nageru.set_channel_signal(2, 0)
+ Nageru.set_channel_signal(3, 1)
Here, channels 2 and 3 (the two first ones) correspond directly to inputs
0 and 1, respectively. The others don't, and return -1. The effect on the
UI is that the user can right-click on the channel and configure the input
that way; in fact, this is currently the only way to configure them.
-Furthermore, channels can have a color::
+Furthermore, channels can have a color, which is governed by Nageru calling
+a function your theme::
function channel_color(channel)
channel when asked; it can vary from frame to frame. A typical use is to mark
the currently playing input as red, or the preview as green.
-And finally, there are two entry points related to white balance::
- function supports_set_wb(channel)
- function set_wb(channel, red, green, blue)
+.. _white-balance:
+
+White balance
+.............
+
+Finally, there are two entry points related to white balance. The first one
+is::
-If the first function returns true (called once, at the start of the program),
+ Nageru.set_supports_wb(2, true)
+
+If the first function is called with a true value (at the start of the theme),
the channel will get a “Set WB” button next to it, which will activate a color
-picker. When the user picks a color (ostensibly with a gray point), the second
+picker, to select the gray point. To actually *apply* this white balance change,
+you have two options. If you're using Nageru 1.9.2 or newer, it's as simple
+as adding one element to the scene::
+
+ scene:add_white_balance()
+
+The white balance effect will automatically figure out which input it is
+connected to, and fetch its gray point if needed. (If it is connected to
+e.g. a mix of several inputs, such as a camera and an overlay, you will need to
+give the input to fetch white balance from as as a parameter.)
+
+If, on the other hand, you are using Nageru 1.9.1 or older (or just wish
+for more manual control), there's an entry point you will need to implement::
+
+ function set_wb(channel, red, green, blue)
+
+When the user picks a gray point, this function
function will be called (with the RGB values in linear light—not sRGB!),
and the theme can then use it to adjust the white balance for that channel.
The typical way to to this is to have a *WhiteBalanceEffect* on each input
and set its “neutral_color” parameter using the “set_vec3” function.
-More complicated channels: Scenes
----------------------------------
+More complicated channels: Composites
+-------------------------------------
-Direct inputs are not the only kind of channels possible; again, any chain
-can be output. The most common case is different kinds of **scenes**,
+Direct inputs are not the only kind of channels possible; again, any scene
+can be output. The most common case is different kinds of **composites**,
typically showing side-by-side or something similar. The typical UI presented
to the user in this case is that you create a channel that consists of the
-finished setup; you use ResampleEffect (or ResizeEffect for low-quality chains),
+finished setup; you use ResampleEffect (or ResizeEffect for preview scenes),
*PaddingEffect* (to place the rectangles on the screen, one of them with a
transparent border) and then *OverlayEffect* (to get both on the screen at
the same time). Optionally, you can have a background image at the bottom,
and perhaps a logo at the top. This allows the operator to select a pre-made
-scene, and then transition to and from it from a single camera view (or even
-between different scenes) as needed.
+composits, and then transition to and from it from a single camera view (or even
+between different composites) as needed.
-Transitions involving scenes tend to be the most complicated parts of the theme
+Transitions involving composites tend to be the most complicated parts of the theme
logic, but also make for the most distinct parts of your visual look.
------------
In addition to video inputs, Nageru supports static **image inputs**.
-These work pretty much the same way as live video inputs; however,
-they need to be instantiated in a different way. Recall that live inputs
-were created like this::
+These work pretty much the same way as live video inputs. Recall that
+you chose what input to display like this::
- input = chain:add_live_input(false, deint)
+ input:display(0)
Image inputs are instead created by instantiating *ImageInput* and
-adding them manually to the chain::
+displaying that::
- input = chain:add_effect(ImageInput.new("bg.jpeg"))
-
-Note that add_effect returns its input for convenience.
+ bg = ImageInput.new("bg.jpeg") -- Once, at the start of the program.
+ input:display(bg) -- In get_scene().
All image types supported by FFmpeg are supported; if you give in a video,
only the first frame is used. The file is checked once every second,
a restart. (If the file contains an error, the update will be ignored.)
This allows you to e.g. have simple message overlays that you can change
without restarting Nageru.
+
+
+.. _locking:
+
+Locking alternatives
+--------------------
+
+In some cases, Nageru may be building in alternatives to a scene that you
+don't really need, resulting in combinatorial explosion. (If the number of
+instances is getting high, you will get a warning when finalizing the scene.)
+For instance, in some cases, you know that a given transition scene will never
+be used for previews, just live. In this case, you can replace the call to
+scene:finalize() with::
+
+ scene:finalize(false)
+
+In this case, you guarantee that the scene will never be returned when
+get_scene() is called with the number 0. (Similarly, you can use true
+to *only* use it for the live channel.)
+
+Similarly, inputs can hold four different input types, but in some scenes,
+you may always use them with a specific one, e.g. an image “bg_img”. In this case,
+you may add the input with a specific type right away::
+
+ scene:add_input(bg_img)
+
+Similarly, for a live input, you can do::
+
+ scene:add_input(0)
+
+You can still use scene:display() to change the input, but it needs to be of
+the same *type* as the one you gave to add_input().
+
+Finally, you can specify that some effects only make sense together, reducing
+the number of possibilities further. For instance, you may have an optional
+crop effect followed by a resample, where the resample is only enabled if the
+crop is. If so, you can do this::
+
+ resample_effect:always_disable_if_disabled(crop_effect)
+
+Also, since Nageru 1.9.1, you can disable an optional effect if a given other
+effect is *enabled*::
+
+ overlay1_effect:promise_to_disable_if_enabled(overlay2_effect)
+ overlay2_effect:promise_to_disable_if_enabled(overlay1_effect)
+
+Note that the latter is a promise from the user, not automatic disabling; since
+it is mostly useful for mutual exclusions, Nageru wouldn't know which of the
+two to disable. (If you violate the promise, you will get an error message at
+runtime.) It can still be useful for reducing the number of alternatives, though.
+
+For more advanced exclusions, you may choose to split up the scenes into several
+distinct ones that you manage yourself; indeed, before Nageru 1.9.0, that was
+the only option. At some point, however, you may choose to simply accept the
+added startup time and a bit of extra RAM cost; ease of use and flexibility often
+trumps such concerns.
+
+
+.. _menus:
+
+Theme menus
+-----------
+
+Complicated themes, especially those dealing with :doc:`HTML inputs <html>`,
+may have needs for user control that go beyond those of transition buttons.
+(An obvious example may be “reload the HTML file”.) For this reason,
+themes can also set simple *theme menus*, which are always visible
+no matter what inputs are chosen.
+
+If a theme chooses to set a theme menu, it will be available on the
+main menu bar under “Theme”; if not, it will be hidden. You can set
+the menu at startup or at any other point, using a simple series of
+labels and function references::
+
+ function modify_aspect()
+ -- Your code goes here.
+ end
+
+ function reload_html()
+ html_input:reload()
+ end
+
+ ThemeMenu.set(
+ { "Change &aspect", modify_aspect },
+ { "&Reload overlay", reload_html }
+ )
+
+When the user chooses a menu entry, the given Lua function will
+automatically be called. There are no arguments nor return values.
+
+Menus can contain submenus, by giving an array instead of a function::
+
+ ThemeMenu.set(
+ { "Overlay", {
+ { "Version A", select_overlay_a },
+ { "Version B", select_overlay_b }
+ },
+ { "&Reload overlay", reload_html }
+ )
+
+They can also be checkable, or have checkboxes, by adding a third
+array element containing flags for that::
+
+ ThemeMenu.set(
+ { "Enable overlay", enable_overlay, Nageru.CHECKED }, -- Currently checked.
+ { "Enable crashing", make_unstable, Nageru.CHECKABLE } -- Can be checked, but isn't currently.
+ )
+
+When such an option is selected, you probably want to rebuild the menu to
+reflect the new state.
+
+There currently is no support for input boxes, sliders,
+or the likes. However, do note that since the theme is written in unrestricted
+Lua, so you can use e.g. `lua-http <https://github.com/daurnimator/lua-http>`_
+to listen for external connections and accept more complicated inputs
+from those.
+
+
+.. _signal-info:
+
+Signal information queries
+--------------------------
+
+As previously mentioned, get_scene() takes in a “signals” parameter
+that you can query for information about each signal (numbered from 0;
+live and preview are channels, not signals), like its current resolution
+or frame rate:
+
+ * get_frame_width(signal), get_frame_height(signal): Width and height of the last frame.
+ * get_width(signal), get_height(signal): Width and height of the last *field*
+ (the field height is half of the frame height for an interlaced signal).
+ * get_interlaced(signal): Whether the last frame was interlaced.
+ * get_has_signal(signal): Whether there is a valid input signal.
+ * get_is_connected(signal): Whether there is even a card connected
+ to this signal (USB cards can be swapped in or out); if not,
+ you will get a stream of single-colored frames.
+ * get_frame_rate_nom(signal), get_frame_rate_den(signal): The frame rate
+ of the last frame, as a rational (e.g. 60/1, or 60000/1001 for 59.94).
+ * get_last_subtitle(signal): See :ref:`subtitle-ingest`.
+ * get_human_readable_resolution(signal): The resolution and frame rate in
+ human-readable form (e.g. “1080i59.94”), suitable for e.g. stream titles.
+ Note that Nageru does not follow the EBU recommendation of using
+ frame rate even for interlaced signals (e.g. “1080i25” instead of “1080i50”),
+ since it is little-used and confusing to most users.
+
+You can use this either for display purposes, or for choosing the right
+effect alternatives. In particular, you may want to disable scaling if
+the frame is already of the correct resolution.
+
+
+Overriding the status line
+--------------------------
+
+Some users may wish to override the status line, e.g. with recording time.
+If so, it is possible (since Nageru 1.9.1) to declare a function **format_status_line**::
+
+ function format_status_line(disk_space_text, file_length_seconds)
+ if file_length_seconds > 86400.0 then
+ return "Time to make a new segment"
+ else
+ return "Disk space left: " .. disk_space_text
+ end
+ end
+
+As demonstrated, it is given the disk space text (that would normally
+be there), and the length of the current recording file in seconds.
+HTML is supported.
projects / nageru-docs / blobdiff