4 In Nageru, most of the business logic around how your stream
5 ends up looking is governed by the **theme**, much like how a
6 theme works on a blog or a CMS. Most importantly, the theme
7 governs the look and feel through the different scenes and
8 transitions between them, such that an event gets a consistent
9 visual language even if the operators differ. Instead of requiring the user
10 to modify Nageru's C++ core, themes are written in
11 `Lua <https://www.lua.org/>`_, a lightweight scripting language
14 Themes contain a lot of logic, and writing one can seem a bit
15 daunting at first. However, most events will be happy just tweaking
16 one of the themes included with Nageru, and the operator (if different
17 from the visual designer) will not need to worry at all.
19 Nageru ships with two themes, a default full-featured two-camera
20 setup with side-by-side for e.g. conferences, and a minimal one
21 that is easier for new users to understand.
24 Introduction to scenes
25 ----------------------
27 Anything that's shown on the stream, or on one of the preview displays,
28 is created by a **Movit chain**, instantiated by a Nageru **scene**.
29 `Movit <https://movit.sesse.net/>`_
30 is a library for high-quality, high-performance video filters,
31 and Nageru's themes can use a simplified version of Movit's API where
32 most of the low-level details are abstracted away.
34 Every frame, the theme chooses a **scene** and a set of parameters to it,
35 based on what it thinks the picture should look like. Every scene
36 consists of a set of *inputs* (which can be either live video streams
37 or static pictures) and then a set of operators or *effects* to combine
38 or modify each other. Movit compiles these down to a set of shaders
39 that run in high speed on the GPU; the theme doesn't see a pixel,
40 and thus, Lua's performance (even though good for its language class,
41 especially if you use `LuaJIT <http://luajit.org/>`_) will not matter
48 The simplest possible scene takes only in an input and sends it on
49 to the display (the output of the last added node is always sent to
50 the screen, and in this case, that would be the input)::
52 local scene = Scene.new(16, 9) -- Aspect ratio.
53 local input = scene:add_input()
54 input:display(0) -- First input card. Can be changed whenever you want.
57 The live scene is always processed in full resolution (typically 720p)
58 and then scaled down for the GUI. Preview scenes are rendered in exactly
59 the resolution required, although of course, intermediate steps could be
63 Setting parameters, and the get_scene entry point
64 -------------------------------------------------
66 Many effects support parameters that can vary per-frame. Imagine,
67 for instance, a theme where you want to supports two inputs and fading between
68 them. This means you will need a scene that produces two inputs and
69 produces a mix of them; Movit's *MixEffect* is exactly what you want here::
71 local scene = EffectChain.new(16, 9)
73 local input0 = scene:add_input()
75 local input1 = scene:add_input()
78 -- Note that add_effect returns its input for convenience.
79 local mix_effect = scene:add_effect(MixEffect.new(), input0, input1)
82 Every frame, Movit will call your **get_scene** function, which has
85 function get_scene(num, t, width, height, signals)
87 “width” and “height” are what you'd expect (the output resolution).
88 t contains the current stream time in seconds. “num” contains 0
89 for the live view, 1 for the preview view, and 2, 3, 4, … for each
90 of the individual stream previews. “signals” contains a bit of
91 information about each input signal (see :ref:`signal-info`).
93 get_scene in return should return a scene. However, before you do that,
94 you can set the parameters **strength_first** and **strength_second**
95 on it; for instance like this::
97 function get_scene(num, t, width, height, signals)
98 -- Assume num is 0 here; you will need to handle the other
100 local fade_progress = 0.0
101 if t >= 1.0 and t >= 2.0: -- Between 1 and 2 seconds; do the fade.
102 fade_progress = t - 1.0
107 mix_effect:set_float("strength_first", 1.0 - fade_progress)
108 mix_effect:set_float("strength_second", fade_progress)
112 Note that in the case where fade_progress is 0.0 or 1.0 (you are just
113 showing one of the inputs), you are wasting GPU power by using the
114 fade scene; you should just return a simpler one-input scene instead.
116 The get_scene function is the backbone of every Nageru theme.
117 As we shall see, however, it may end up dealing with a fair bit
118 of complexity as the theme grows.
121 Scene variants and effect alternatives
122 --------------------------------------
124 Setting up and finalizing a scene is relatively fast, but it still
125 takes a measurable amount of CPU time, since it needs to create an OpenGL
126 shader and have it optimized by the driver; 50–100 ms is not uncommon.
127 Given that 60 fps means each frame is 16.7 ms, you cannot create new scenes in
128 get_scene; every scene you could be using must be created at program start,
129 when your theme is initialized.
131 For any nontrivial theme, there are a lot of possible scenes. Let's
132 return to the case of the MixEffect scene from the previous section.
133 Now let us assume that we could deal with signals that come in at
134 1080p instead of the native 720p. In this case, we will want a high-quality
135 scaler before mixing; *ResampleEffect* provides one::
137 local scene = EffectChain.new(16, 9)
139 local input0 = scene:add_input()
141 local input0_scaled = scene:add_optional_effect(ResampleEffect.new()) -- Implicitly uses input0.
142 input0_scaled:set_int("width", 1280) -- Could also be set in get_scene().
143 input0_scaled:set_int("height", 720)
144 input0_scaled:enable() -- Enable or disable as needed.
146 local input1 = scene:add_input()
148 local input1_scaled = ... -- Similarly here and the rest.
150 input1_scaled:enable_if(some_variable) -- Convenience form for enable() or disable() depending on some_variable.
152 -- The rest is unchanged.
154 Clearly, there are four options here; both inputs could be unscaled,
155 input0 could be scaled but not input1, input1 could be scaled but not input0,
156 or both could be scaled. That means four scenes. However, you don't need to
157 care about this; behind the scenes (no pun intended), Nageru will make all
158 four versions for you and choose the right one as you call enable() or
159 disable() on each effect.
161 Beyond simple on/off switches, an effect can have many *alternatives*,
162 by giving in an array of effects. For instance, it is usually pointless to use
163 the high-quality resampling provided by ResampleEffect for the on-screen
164 outputs; we can use *ResizeEffect* (a simpler scaling algorithm provided
165 directly by the GPU) that instead. The scaling is set up like this::
167 local input0 = scene:add_input()
169 local input0_scaled = scene:add_effect({ResampleEffect.new(), ResizeEffect.new()}) -- Implicitly uses input0.
170 input0_scaled:set_int("width", 1280) -- Just like before.
171 input0_scaled:set_int("height", 720)
173 -- Pick one in get_scene() like this:
174 input0_scaled:choose(ResizeEffect)
176 -- Or by numerical index:
177 input0_scaled:choose(1) -- Chooses ResizeEffect
179 Actually, add_optional_effect() is just a wrapper around add_effect() with
180 IdentityEffect as the other alternative, and disable() is a convenience version of
181 choose(IdentityEffect).
183 All alternatives must
184 have the same amount of inputs, with an exception for IdentityEffect, which can
185 coexist with an effect requiring any amount of inputs (if selected, the IdentityEffect
186 just passes its first input unchanged). Similarly, if you set a parameter with
187 set_int() or similar, it must be valid for all alternatives (again excepting
188 IdentityEffect); if there is one that can only be used on a certain alternative,
189 you must set it directly on the effect::
191 local resample_effect = ResampleEffect.new()
192 resample_effect:set_float("zoom_x", 1.0001) -- Not valid for ResizeEffect.
194 local input0_scaled = scene:add_effect({resample_effect, ResizeEffect.new()})
195 input0_scaled:set_int("width", 1280) -- Set on both alternatives.
196 input0_scaled:set_int("height", 720)
198 -- This is also possible, as choose() returns the chosen effect:
199 input0_scaled:choose(ResampleEffect):set_float("zoom_y", 1.0001)
201 Actually, more versions are created than you'd immediately expect.
202 In particular, the output format for the live output and all previews are
203 different (Y'CbCr versus RGBA), which is also handled transparently for you.
204 Also, the inputs could be interlaced, or they could be images, or videos (see
205 :ref:`images` and :doc:`video`), creating many more options. Again, you
206 generally don't need to care about this; Movit will make sure each and every of
207 those generated scenes runs optimally on your GPU. However, if the
208 combinatorial explosion increases startup time beyond what you are comfortable
209 with, see :ref:`locking`.
215 As we have seen, the theme is king when it determines what to show
216 on screen. However, ultimately, it wants to delegate that power
217 to the operator. The abstraction presented from the theme to the user
218 is in the form of **transitions**. Every frame, Nageru calls the
219 following Lua entry point::
221 function get_transitions(t)
223 (t is again the stream time, but it is provided only for convenience;
224 not all themes would want to use it.) get_transitions must return an array of
225 (currently exactly) three strings, of which any can be blank. These three
226 strings are used as labels on one button each, and whenever the operator clicks
227 one of them, Nageru calls this function in the theme::
229 function transition_clicked(num, t)
231 where “num” is 0, 1 or 2, and t is again the theme time.
233 It is expected that the theme will use this and its internal state
234 to provide the abstraction (or perhaps illusion) of transitions to
235 the user. For instance, a theme will know that the live stream is
236 currently showing input 0 and the preview stream is showing input 1.
237 In this case, it can use two of the buttons to offer “Cut“ or “Fade”
238 transitions to the user. If the user clicks the cut button, the theme
239 can simply switch input and previews, which will take immediate
240 effect on the next frame. However, if the user clicks the fade button,
241 state will need to be set up so that next time get_scene() runs,
242 it will return the scene with the MixEffect, until it determines
243 the transition is over and changes back to showing only one input
244 (presumably the new one).
252 In addition to the live and preview outputs, a theme can declare
253 as many individual **channels** as it wants. These are shown at the
254 bottom of the screen, and are intended for the operator to see
255 what they can put up on the preview (in a sense, a preview of the
258 The number of channels is determined by calling this function
259 once at the start of the program::
261 Nageru.set_num_channels(2)
263 0 is allowed, but doesn't make a lot of sense. Live and preview comes in
266 Each channel will have a label on it; you set it by calling::
268 Nageru.set_channel_name(2, "Side-by-side")
270 Here, channel is 2, 3, 4, etc.—by default, 0 is called “Live” and
271 1 is called “Preview”, and you probably don't need to change this.
273 Each channel has its own scene, starting from number 2 for the first one
274 (since 0 is live and 1 is preview). The simplest form is simply a direct copy
275 of an input, and most themes will include one such channel for each input.
276 (Below, we will see that there are more types of channels, however.)
277 Since the mapping between the channel UI element and inputs is so typical,
278 Nageru allows the theme to simply declare that a channel corresponds to
281 Nageru.set_channel_signal(2, 0)
282 Nageru.set_channel_signal(3, 1)
284 Here, channels 2 and 3 (the two first ones) correspond directly to inputs
285 0 and 1, respectively. The others don't, and return -1. The effect on the
286 UI is that the user can right-click on the channel and configure the input
287 that way; in fact, this is currently the only way to configure them.
289 Furthermore, channels can have a color, which is governed by Nageru calling
290 a function your theme::
292 function channel_color(channel)
294 The theme should return a CSS color (e.g. “#ff0000”, or “cyan”) for each
295 channel when asked; it can vary from frame to frame. A typical use is to mark
296 the currently playing input as red, or the preview as green.
304 Finally, there are two entry points related to white balance. The first one
307 Nageru.set_supports_wb(2, true)
309 If the first function is called with a true value (at the start of the theme),
310 the channel will get a “Set WB” button next to it, which will activate a color
311 picker, to select the gray point. To actually *apply* this white balance change,
312 add the white balance element to the scene::
314 scene:add_white_balance()
316 The white balance effect will automatically figure out which input it is
317 connected to, and fetch its gray point if needed. (If it is connected to
318 e.g. a mix of several inputs, such as a camera and an overlay, you will need to
319 give the input to fetch white balance from as as a parameter.)
322 More complicated channels: Composites
323 -------------------------------------
325 Direct inputs are not the only kind of channels possible; again, any scene
326 can be output. The most common case is different kinds of **composites**,
327 typically showing side-by-side or something similar. The typical UI presented
328 to the user in this case is that you create a channel that consists of the
329 finished setup; you use ResampleEffect (or ResizeEffect for preview scenes),
330 *PaddingEffect* (to place the rectangles on the screen, one of them with a
331 transparent border) and then *OverlayEffect* (to get both on the screen at
332 the same time). Optionally, you can have a background image at the bottom,
333 and perhaps a logo at the top. This allows the operator to select a pre-made
334 composits, and then transition to and from it from a single camera view (or even
335 between different composites) as needed.
337 Transitions involving composites tend to be the most complicated parts of the theme
338 logic, but also make for the most distinct parts of your visual look.
346 In addition to video inputs, Nageru supports static **image inputs**.
347 These work pretty much the same way as live video inputs. Recall that
348 you chose what input to display like this::
352 Image inputs are instead created by instantiating *ImageInput* and
355 bg = ImageInput.new("bg.jpeg") -- Once, at the start of the program.
356 input:display(bg) -- In get_scene().
358 All image types supported by FFmpeg are supported; if you give in a video,
359 only the first frame is used. The file is checked once every second,
360 so if you update the file on-disk, it will be available in Nageru without
361 a restart. (If the file contains an error, the update will be ignored.)
362 This allows you to e.g. have simple message overlays that you can change
363 without restarting Nageru.
371 In some cases, Nageru may be building in alternatives to a scene that you
372 don't really need, resulting in combinatorial explosion. (If the number of
373 instances is getting high, you will get a warning when finalizing the scene.)
374 For instance, in some cases, you know that a given transition scene will never
375 be used for previews, just live. In this case, you can replace the call to
376 scene:finalize() with::
378 scene:finalize(false)
380 In this case, you guarantee that the scene will never be returned when
381 get_scene() is called with the number 0. (Similarly, you can use true
382 to *only* use it for the live channel.)
384 Similarly, inputs can hold four different input types, but in some scenes,
385 you may always use them with a specific one, e.g. an image “bg_img”. In this case,
386 you may add the input with a specific type right away::
388 scene:add_input(bg_img)
390 Similarly, for a live input, you can do::
394 You can still use scene:display() to change the input, but it needs to be of
395 the same *type* as the one you gave to add_input().
397 Finally, you can specify that some effects only make sense together, reducing
398 the number of possibilities further. For instance, you may have an optional
399 crop effect followed by a resample, where the resample is only enabled if the
400 crop is. If so, you can do this::
402 resample_effect:always_disable_if_disabled(crop_effect)
404 Also, you can disable an optional effect if a given other
405 effect is *enabled*::
407 overlay1_effect:promise_to_disable_if_enabled(overlay2_effect)
408 overlay2_effect:promise_to_disable_if_enabled(overlay1_effect)
410 Note that the latter is a promise from the user, not automatic disabling; since
411 it is mostly useful for mutual exclusions, Nageru wouldn't know which of the
412 two to disable. (If you violate the promise, you will get an error message at
413 runtime.) It can still be useful for reducing the number of alternatives, though.
415 For more advanced exclusions, you may choose to split up the scenes into several
416 distinct ones that you manage yourself. At some point, however, you may choose
417 to simply accept the added startup time and a bit of extra RAM cost; ease of
418 use and flexibility often trumps such concerns.
426 Complicated themes, especially those dealing with :doc:`HTML inputs <html>`,
427 may have needs for user control that go beyond those of transition buttons.
428 (An obvious example may be “reload the HTML file”.) For this reason,
429 themes can also set simple *theme menus*, which are always visible
430 no matter what inputs are chosen.
432 If a theme chooses to set a theme menu, it will be available on the
433 main menu bar under “Theme”; if not, it will be hidden. You can set
434 the menu at startup or at any other point, using a simple series of
435 labels and function references::
437 function modify_aspect()
438 -- Your code goes here.
441 function reload_html()
446 { "Change &aspect", modify_aspect },
447 { "&Reload overlay", reload_html }
450 When the user chooses a menu entry, the given Lua function will
451 automatically be called. There are no arguments nor return values.
453 Menus can contain submenus, by giving an array instead of a function::
457 { "Version A", select_overlay_a },
458 { "Version B", select_overlay_b }
460 { "&Reload overlay", reload_html }
463 They can also be checkable, or have checkboxes, by adding a third
464 array element containing flags for that::
467 { "Enable overlay", enable_overlay, Nageru.CHECKED }, -- Currently checked.
468 { "Enable crashing", make_unstable, Nageru.CHECKABLE } -- Can be checked, but isn't currently.
471 When such an option is selected, you probably want to rebuild the menu to
472 reflect the new state.
474 There currently is no support for input boxes, sliders,
475 or the likes. However, do note that since the theme is written in unrestricted
476 Lua, so you can use e.g. `lua-http <https://github.com/daurnimator/lua-http>`_
477 to listen for external connections and accept more complicated inputs
483 Signal information queries
484 --------------------------
486 As previously mentioned, get_scene() takes in a “signals” parameter
487 that you can query for information about each signal (numbered from 0;
488 live and preview are channels, not signals), like its current resolution
491 * get_frame_width(signal), get_frame_height(signal): Width and height of the last frame.
492 * get_width(signal), get_height(signal): Width and height of the last *field*
493 (the field height is half of the frame height for an interlaced signal).
494 * get_interlaced(signal): Whether the last frame was interlaced.
495 * get_has_signal(signal): Whether there is a valid input signal.
496 * get_is_connected(signal): Whether there is even a card connected
497 to this signal (USB cards can be swapped in or out); if not,
498 you will get a stream of single-colored frames.
499 * get_frame_rate_nom(signal), get_frame_rate_den(signal): The frame rate
500 of the last frame, as a rational (e.g. 60/1, or 60000/1001 for 59.94).
501 * get_last_subtitle(signal): See :ref:`subtitle-ingest`.
502 * get_human_readable_resolution(signal): The resolution and frame rate in
503 human-readable form (e.g. “1080i59.94”), suitable for e.g. stream titles.
504 Note that Nageru does not follow the EBU recommendation of using
505 frame rate even for interlaced signals (e.g. “1080i25” instead of “1080i50”),
506 since it is little-used and confusing to most users.
508 You can use this either for display purposes, or for choosing the right
509 effect alternatives. In particular, you may want to disable scaling if
510 the frame is already of the correct resolution.
516 Before you attempt to control audio from the theme, be sure to have read
517 the documentation about :doc:`audio`.
519 The theme has a certain amount of control over the audio
520 mix, assuming that you are in multichannel mode. This is useful in particular
521 to be able to set defaults, if e.g. one channel should always be muted at
522 startup, or to switch in/out certain channels depending on whether they are
525 In particular, these operations are available::
527 # Returns number of buses in the mapping.
528 local num_buses = Nageru.get_num_audio_buses()
530 # Gets the name from the mapping. All indexes start at zero,
531 # so valid indexes are 0..(num_buses-1), inclusive.
532 local name = Nageru.get_audio_bus_name(N)
534 # 0.0 is zero amplification, as in the UI. Valid range is
535 # -inf to +6.0, inclusive.
536 local level = Nageru.get_audio_bus_fader_level_db(N)
537 set_audio_bus_fader_level_db(N, level)
539 # Similar as the above, but valid range is -15.0..+15.0 (dB).
540 # Valid bands are Nageru.EQ_BAND_{BASS, MID, TREBLE}.
541 local eq_level = Nageru.get_audio_bus_eq_level_db(N, Nageru.EQ_BAND_BASS)
542 Nageru.set_audio_bus_eq_level_db(N, Nageru.EQ_BAND_BASS, level)
544 # A boolean. Does not affect the bus levels set/returned above.
545 local muted = Nageru_get_audio_bus_mute(N)
546 Nageru_set_audio_bus_mute(N, false)
548 Note that any audio operation is inherently unsynchronized with the UI,
549 so if the user reduces the number of audio buses while
550 the theme tries to access one that is going away, you may get unpredictable
551 behavior, up to and including crashes. Thus, you will need to be careful
552 with such operations.
554 Also, you cannot do anything with audio before the first *get_scene()* call,
555 since the audio mixer is initialized only after the theme has been loaded and
556 initialized. Thus, for things that should be done only the first frame, the
557 recommended method is to put code into get_scene() and have a guard variable
558 that makes sure it is only run
562 Overriding the status line
563 --------------------------
565 Some users may wish to override the status line, e.g. with recording time.
566 If so, it is possible to declare a function **format_status_line**::
568 function format_status_line(disk_space_text, file_length_seconds)
569 if file_length_seconds > 86400.0 then
570 return "Time to make a new segment"
572 return "Disk space left: " .. disk_space_text
576 As demonstrated, it is given the disk space text (that would normally
577 be there), and the length of the current recording file in seconds.