From 619d8a89f87a555aabdedb9c60a9ac2902fca18a Mon Sep 17 00:00:00 2001 From: "Steinar H. Gunderson" Date: Mon, 7 Nov 2016 21:57:24 +0100 Subject: [PATCH] Add a node about writing themes. --- index.rst | 1 + theme.rst | 342 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 343 insertions(+) create mode 100644 theme.rst diff --git a/index.rst b/index.rst index a53411b..8dac857 100644 --- a/index.rst +++ b/index.rst @@ -12,6 +12,7 @@ Contents: meintro hardware streaming + theme Indices and tables diff --git a/theme.rst b/theme.rst new file mode 100644 index 0000000..655329e --- /dev/null +++ b/theme.rst @@ -0,0 +1,342 @@ +The theme +========= + +In Nageru, most of the business logic around how your stream +ends up looking is governed by the **theme**, much like how a +theme works on a blog or a CMS. Most importantly, the theme +governs the look and feel through the different scenes and +transitions between them, such that an event gets a consistent +visual language even if the operators differ. Instead of requiring the user +to modify Nageru's C++ core, themes are written in +`Lua `_, a lightweight scripting language +made for embedding. + +Themes contain a lot of logic, and writing one can seem a bit +daunting at first. However, most events will be happy just tweaking +one of the themes included with Nageru, and the operator (if different +from the visual designer) will not need to worry at all. + +Nageru ships with two themes, a default full-featured two-camera +setup with side-by-side for e.g. conferences, and a minimal one +that is easier for new users to understand. + + +Introduction to chains +---------------------- + +Anything that's shown on the stream, or on one of the preview displays, +is created by a **Movit chain**. `Movit `_ +is a library for high-quality, high-performance video filters, +and Nageru's themes can use a simplified version of Movit's API where +most of the low-level details are abstracted away. + +Every frame, the theme choses a chain and a set of parameters to it, +based on what it thinks the picture should look like. Every chain +consists of a set of *inputs* (which can be either live video streams +or static pictures) and then a set of operators or *effects* to combine +or modify each other. Movit compiles these down to a set of shaders +that run in high speed on the GPU; the theme doesn't see a pixel, +and thus, Lua's performance (even though good for its language class, +especially if you use `LuaJIT `_) will not matter +much. + + +High- and low-quality chains +---------------------------- + +The simplest possible chain takes only in an input and sends it on +to the display (the output of the last added node is always sent to +the screen, and in this case, that would be the input):: + + local chain = EffectChain.new(16, 9) -- Aspect ratio. + local input = chain:add_live_input(false, false) -- No bounce override, no deinterlacing. + input:connect_signal(0) -- First input card. Can be changed whenever you want. + chain:finalize(hq) + +Note the “hq” parameter. Every chain needs to be able to run in two +situations: Both for the stream output (the ”live” pane) and for the preview +displays (the big “preview” pane and the smaller ones). The details have +to do with Nageru internals (high-quality chains need to have an additional +Y'CbCr output), but the distinction is also useful for themes. In particular, +some operations, like scaling, can be done in various quality levels, +and for a low-resolution preview display, you don't need maximum quality. +Thus, in a preview chain (hq=false), you can safely take shortcuts. + +The live chain is always processed in full resolution (typically 720p) +and then scaled down for the GUI. Preview chains are rendered in exactly +the resolution required, although of course, intermediate steps could be +bigger. + +Setting parameters, and the get_chain entry point +------------------------------------------------- + +Many effects support parameters that can vary per-frame. Imagine, +for instance, a theme where you want to supports two inputs and fading between +them. This means you will need a chain that produces two inputs and +produces a mix of them; Movit's *MixEffect* is exactly what you want here:: + + local chain = EffectChain.new(16, 9) + + local input0 = chain:add_live_input(false, false) + input0:connect_signal(0) + local input1 = chain:add_live_input(false, false) + input1:connect_signal(1) + + local mix_effect = chain:add_effect(MixEffect.new(), input0, input1) + chain:finalize(hq) + +Every frame, Movit will call your **get_chain** function, which has +this signature: + + function get_chain(num, t, width, height, signals) + +“width” and “height” are what you'd expect (the output resolution). +t contains the current stream time in seconds. “num” contains 0 +for the live view, 1 for the preview view, and 2, 3, 4, … for each +of the individual stream previews. “signals“ contains a bit of +information about each input signal, like its current resolution +or frame rate. + +get_chain is in turn responsible for returning two values: + + * The first return value is a Movit chain, as described in these + sections. For the live stream (num=0), you should return a high-quality + chain; for all others, you should return a low-quality chain. + * The second parameter is an *closure* that will be called just before + the chain is to be rendered. (The same chain could be used in + multiple OpenGL contexts at the same time, so you can't just set the values + immediately before returning. If you set them in the closure, + Nageru and Movit will deal with all the required threading for you.) + +In the returned closure, you can set the parameters **strength_first** +and **strength_second**; for instance like this:: + + function get_chain(num, t, width, height, signals) + -- Assume num is 0 here; you will need to handle the other + -- cases, too. + prepare = function() + input0:connect_signal(0) + input1:connect_signal(1) + + local fade_progress = 0.0 + if t >= 1.0 and t >= 2.0: -- Between 1 and 2 seconds; do the fade. + fade_progress = t - 1.0 + elseif t >= 2.0: + fade_progress = 1.0 + end + + mix_effect:set_float("strength_first", 1.0 - fade_progress) + mix_effect:set_float("strength_second", fade_progress) + end + return chain, prepare + end + +Note that in the case where fade_progress is 0.0 or 1.0 (you are just +showing one of the inputs), you are wasting GPU power by using the +fade chain; you should just return a simpler one-input chain instead. + +The get_chain function is the backbone of every Nageru theme. +As we shall see, however, it may end up dealing with a fair bit +of complexity as the theme grows. + + +Chain precalculation +-------------------- + +Setting up and finalizing a chain is relatively fast, but it still +takes a measurable amount of CPU time, since it needs to create an OpenGL +shader and have it optimized by the driver; 50–100 ms is not uncommon. +Given that 60 fps means each frame is 16.7 ms, you cannot create new chains in +get_chain; every chain you could be using must be created at program start, +when your theme is initialized. + +For any nontrivial theme, there are a lot of possible chains. Let's +return to the case of the MixEffect chain from the previous section. +Now let us assume that we could deal with signals that come in at +1080p instead of the native 720p. In this case, we will want a high-quality +scaler before mixing; *ResampleEffect* provides one:: + + local chain = EffectChain.new(16, 9) + + local input0 = chain:add_live_input(false, false) + input0:connect_signal(0) + local input0_scaled = chain:add_effect(ResampleEffect.new()) -- Implicitly uses input0. + chain_or_input.resample_effect:set_int("width", 1280) -- Would normally be set in the prepare function. + chain_or_input.resample_effect:set_int("height", 720) + + local input1 = chain:add_live_input(false, false) + input1:connect_signal(1) + + -- The rest is unchanged. + +Clearly, there are four options here; both inputs could be unscaled, +input0 could be scaled but not input1, input1 could be scaled but not input0, +or both could be scaled. That means four chains. + +Now remember that we need to create all your chains both in high- +and low-quality versions. In particular, this determines the “hq” +parameter to finalize(), but in our case, we would want to replace +ResampleEffect by *ResizeEffect* (a simpler scaling algorithm provided +directly by the GPU) for the low-quality versions. This makes for +eight chains. + +Now also consider that we would want to deal with *interlaced* +inputs. (You can check if you get an interlaced input on the Nth +input by calling “signals:get_deinterlaced(n)” from get_chain.) +This further quadruples the number of chains you'd need to write, +and this isn't even including that you'd want the static chains. +It is obvious that this should not be done by hand. The default +included theme contains a handy Lua shortcut called +**make_cartesian_product** where you can declare all the dimensions +you would want to specialize your chain over, and have a callback +function called for each possible combination. Movit will make sure +each and every of those generated chains runs optimally on your GPU. + + +Transitions +----------- + +As we have seen, the theme is king when it determines what to show +on screen. However, ultimately, it wants to delegate that power +to the operator. The abstraction presented from the theme to the user +is in the form of **transitions**. Every frame, Nageru calls the +following Lua entry point:: + + function get_transitions(t) + +(t is again the stream time, but it is provided only for convenience; +not all themes would want to use it.) get_transitions must return an array of +(currently exactly) three strings, of which any can be blank. These three +strings are used as labels on one button each, and whenever the operator clicks +one of them, Nageru calls this function in the theme:: + + function transition_clicked(num, t) + +where “num” is 0, 1 or 2, and t is again the theme time. + +It is expected that the theme will use this and its internal state +to provide the abstraction (or perhaps illusion) of transitions to +the user. For instance, a theme will know that the live stream is +currently showing input 0 and the preview stream is showing input 1. +In this case, it can use two of the buttons to offer “Cut“ or “Fade” +transitions to the user. If the user clicks the cut button, the theme +can simply switch input and previews, which will take immediate +effect on the next frame. However, if the user clicks the fade button, +state will need to be set up so that next time get_chain() runs, +it will return the chain with the MixEffect, until it determines +the transition is over and changes back to showing only one input +(presumably the new one). + + +Channels +-------- + +In addition to the live and preview outputs, a theme can declare +as many individual **channels** as it wants. These are shown at the +bottom of the screen, and are intended for the operator to see +what they can put up on the preview (in a sense, a preview of the +preview). + +The number of channels is determined by calling this function +once at the start of the program:: + + function num_channels() + +It should simply return the number of channels (0 is allowed, +but doesn't make a lot of sense). Live and preview comes in addition to this. + +Each channel will have a label on it; Nageru asks the theme +by calling this function:: + + function channel_name(channel) + +Here, channel is 2, 3, 4, etc.—0 is always called “Live” and +1 is always called “Preview”. + +Each channel has its own chain, starting from number 2 for the first one +(since 0 is live and 1 is preview). The simplest form is simply a direct copy +of an input, and most themes will include one such channel for each input. +(Below, we will see that there are more types of channels, however.) +Since the mapping between the channel UI element and inputs is so typical, +Nageru allows the theme to simply declare that a channel corresponds to +a given signal, by asking it:: + + function channel_signal(channel) + if channel == 2 then + return 0 + elseif channel == 3 then + return 1 + else + return -1 + end + end + +Here, channels 2 and 3 (the two first ones) correspond directly to inputs +0 and 1, respectively. The others don't, and return -1. The effect on the +UI is that the user can right-click on the channel and configure the input +that way; in fact, this is currently the only way to configure them. + +Furthermore, channels can have a color:: + + function channel_color(channel) + +The theme should return a CSS color (e.g. “#ff0000”, or “cyan”) for each +channel when asked; it can vary from frame to frame. A typical use is to mark +the currently playing input as red, or the preview as green. + +And finally, there are two entry points related to white balance:: + + function supports_set_wb(channel) + function set_wb(channel, red, green, blue) + +If the first function returns true (called once, at the start of the program), +the channel will get a “Set WB” button next to it, which will activate a color +picker. When the user picks a color (ostensibly with a gray point), the second +function will be called (with the RGB values in linear light—not sRGB!), +and the theme can then use it to adjust the white balance for that channel. +The typical way to to this is to have a *WhiteBalanceEffect* on each input +and set its “neutral_color” parameter using the “set_vec3” function. + + +More complicated channels: Scenes +--------------------------------- + +Direct inputs are not the only kind of channels possible; again, any chain +can be output. The most common case is different kinds of **scenes**, +typically showing side-by-side or something similar. The typical UI presented +to the user in this case is that you create a channel that consists of the +finished setup; you use ResampleEffect (or ResizeEffect for low-quality chains), +*PaddingEffect* (to place the rectangles on the screen, one of them with a +transparent border) and then *OverlayEffect* (to get both on the screen at +the same time). Optionally, you can have a background image at the bottom, +and perhaps a logo at the top. This allows the operator to select a pre-made +scene, and then transition to and from it from a single camera view (or even +between different scenes) as needed. + +Transitions involving scenes tend to be the most complicated parts of the theme +logic, but also make for the most distinct parts of your visual look. + + +Image inputs +------------ + +In addition to video inputs, Nageru supports static **image inputs**. +These work pretty much the same way as live video inputs; however, +they need to be instantiated in a different way. Recall that live inputs +were created like this:: + + input = chain:add_live_input(false, deint) + +Image inputs are instead created by instantiating **ImageInput** and +adding them manually to the chain:: + + input = chain:add_effect(ImageInput.new("bg.jpeg")) + +Note that add_effect returns its input for convenience. + +All image types supported by FFmpeg are supported; if you give in a video, +only the first frame is used. The file is checked once every second, +so if you update the file on-disk, it will be available in Nageru without +a restart. (If the file contains an error, the update will be ignored.) +This allows you to e.g. have simple message overlays that you can change +without restarting Nageru. -- 2.39.2