It is strongly recommended to run Futatabi on a separate machine from Nageru,
not the least because you probably want a different person to operate the replay
-while the producer is operating Nageru.
+while the producer is operating Nageru. (See :ref:`coop`.)
Like Nageru, Futatabi uses your GPU for nearly all image processing.
However, unlike Nageru, Futatabi requires a powerful GPU for interpolation;
Getting started
---------------
+Futatabi always pulls data from Nageru over the network; it doesn't support SDI
+input or output. Assuming you have a recent version of Nageru (typically one
+that comes with Futatabi), it is capable of sending all of its cameras as one
+video stream (see :ref:`futatabiformat`), so you can start Futatabi with
+
+ ./futatabi http://nageru-server.example.org/multicam.mp4
+
+If you do not have a running Nageru installation, see :ref:`sampledata`.
+
+Once you are up and connected, Futatabi will start recording all of the frames
+to disk. This process happens continuously for as long as you have disk space,
+even if you are playing something back or editing clips, and the streams are
+displayed in real time in mini-display as they come in. You make replays in
+the form of *clips* (the top list is the clip list), which are then queued into
+the *playlist* (the bottom list). Your end result is a live HTTP stream that
+can be fed back into Nageru as a video input; by default, Futatabi listens
+on port 9096.
+
+Futatabi has the concept of a *workspace*, which defaults to your current
+directory (you can change it with the -d option). This holds all of the
+recorded frames, all metadata about clips, and preferences set from the menus
+(interpolation quality and cue point padding). If you quit Futatabi and
+restart it (or you go down as the result of a power failure or the likes),
+it will remember all the state and frames for you. As a special case,
+if you give */dev/null* in place of an input URL or file, you can keep using
+your workspace without getting any new frames added.
+
+.. _basicui:
+
+Basic UI operations
+'''''''''''''''''''
+
+Nageru can be operated using either the keyboard and mouse, or using a MIDI
+controller with some help of the mouse. In this section, we will be discussing
+the keyboard and mouse only; see :ref:`midi` for details on using MIDI
+controllers.
+
+A clip in the clip list consists simply of an in and an out point; it represents
+an interval of physical time (although timed to the video clock). A clip in the
+playlist contains the same information plus some playback parameters, in particular
+which camera (stream) to play back.
+
+Clicking the ”Cue in” button, or pressing the A key, will start a new clip in
+the clip list that begins at the current time. Similarly, clicking the ”Cue
+out” button, or pressing the S key, will set the end point for said clip.
+(If you click a button twice, it will overwrite the previous result.)
+Now it is in a state where you can queue it to the play list (mark the camera
+you want to use and click the “Queue” button, or Q on the keyboard) for
+playing, or you can preview them using the “Preview” button (W on the
+keyboard). Previewing can be done either from the clip list the clip list or
+the playlist; they will not be interpolated or faded, but will be played back
+in the right
+
+You can edit cue points, both in the clip and the playlist, in two ways:
+Either use the scroll wheel on the mouse, or hold down the left mouse button
+and scrub left or right. (You can hold down Shift to change ten times as fast,
+or Alt to change at one-tenth the speed.) You'll see the new cue points as you
+change them in the preview window. You can also give clips names; these don't
+mean anything, but can be good references to locate a specific kind of clip
+for later use. Once you're happy with a playlist, and your producer is ready
+to cut to your channel, click on the first clip you want to play back and
+click the “Play” button (space on the keyboard); the result will both be
+visible in the top screen and go out live over the network to Nageru.
+
+.. _speed:
+
+Controlling the playback speed
+''''''''''''''''''''''''''''''
+
+.. _coop:
+
+Working with your producer
+''''''''''''''''''''''''''
+
+Generally, we recommend that the producer (Nageru operator) and slow motion
+operator (Futatabi operator) sit closely together and can communicate verbally.
+Good cooperation between the two is essential to get a good final product;
+especially the switches to and from the replays can be a critical point.
+
+The general rule for working together is fairly obvious: The producer should
+switch to a replay when there's something to show, and switch away when there's
+nothing more to show (or, less ideally, when something live takes priority).
+Generally, when you have a playlist ready, inform your producer; they will
+count you in (three, two, one, go). At one, start playing so that you have some
+margin. If the Nageru theme is set up correctly (see :ref:`talkback`), they will
+know how much is queued up so that they can switch back before the replay runs
+out, but it doesn't hurt to give a warning up-front. The producer might also
+be able to request replays of specific events, or ask you to save something
+for later if they can't show it right now (e.g. a foul situation that wasn't called).
+
+Replay workflows
+----------------
+
+On top of the basics outlined in :ref:`basicui`, there are many possible
+workflows; we'll discuss only two. Try out a few and see which ones fit your
+style and type of event.
+
+Repeated cue-in
+'''''''''''''''
+
+In many sports, you don't necessarily know when a replay-worthy event has happened
+before it's already happened. However, you may reasonably know when something is
+*not* happening, and it would be a good time to start a clip if something is happening
+immediately afterwards. At these points, you can make repeated cue-ins, ie., start
+a clip without finishing it. As long as you keep making cue-ins, the previous one
+will be discarded. Once you see that something *is* happening, you can wait until
+it's done, and then do a cue-out, which gives you a good clip immediately.
+
+For instance, in a basketball game, you could be seeing a series of uninteresting
+passes, clicking cue-in on each of them. However, once it looks like there's an
+opportunity for a score, you can hold and see what happens; if the shot happens,
+you can click cue-out, and if not, you can go back.
+
+Before playing the clip, you can make adjustments to the in and out points
+as detailed above. This will help you trim away any uninteresting lead-ups,
+or add more margins for fades. If you consistently find that you have too
+little margin, you can use the *cue point padding* feature (either from the
+command line using *--cue-point-padding*, or set from the menu). If you set
+cue point padding to e.g. two seconds, the cue-in point will automatically be set
+two seconds ago when you cue-in, and the cue-out point will be set two seconds
+into the future when you cue-out.
+
+
+Instant clips
+'''''''''''''
+
+Like the previous section explained how you generally would know the *start*
+of an interesting event (at least if discarding most of the candidates),
+you would be even more sure about the *end* of one. Thus, you can simply wait
+until something interesting has happened, and then click cue-in immediately
+followed by cue-out. This will give you a clip of near zero length, ending
+at the right point. Then, edit this clip to set the starting point as needed,
+and it's ready to play.
+
+Again, you can use the cue point padding feature to your advantage; if so,
+your clips will not be of zero length, but rather of some predefined length
+given by your chosen cue point padding.
+
+
+.. _sampledata:
+
Sample multicamera data
'''''''''''''''''''''''
on the camera operators. In short, they should be realistic input material
to practice with.
-Please download these files only once, instead of streaming them directly over
-HTTP each time you want to test.
-
Transferring data to and from Nageru
------------------------------------
Video format specification
''''''''''''''''''''''''''
+Futatabi expects to get data in MJPEG format only; though MJPEG is old,
+it yields fairly good quality per bit for an intraframe format, supports
+4:2:2 without too many issues, and has hardware support through VA-API
+for both decode (since Ivy Bridge) and encode (since Skylake). The latter
+is especially important for Futatabi, since there are so many high-resolution
+streams; software encode/decode of several 1080p60 streams at the same time
+is fairly taxing on the CPU if done in software. This means we can easily
+send 4:2:2 camera streams back and forth between Nageru and Futatabi without having
+to scale or do other lossy processing (except of course the compression itself).
+
+However, JPEG as such does not have any way of specifying things like color
+spaces and chroma placement. JFIF, the *de facto* JPEG standard container,
+specifies conventions that are widely followed, but they do not match what
+comes out of a capture card. Nageru's multicam export _does_ set the appropriate
+fields in the output Matroska mux (which is pretty much the only mux that can
+hold such information), but there are few if any programs that read them and give
+them priority over JFIF's defaults. Thus, if you want to use the multicam stream
+for something other than Futatabi, or feed Futatabi with data not from Nageru,
+there are a few subtle issues to keep in mind.
+
+In particular:
+
+ * Capture cards typically send limited-range Y'CbCr (luma between 16..235
+ and chroma between 16..240); JFIF is traditionally full-range (0..255
+ for both). (See also :ref:`synthetictests`.) Note that there is a special
+ private JPEG comment added to signal this, which FFmpeg understands.
+ * JFIF, like MPEG, assumes center chroma placement; capture cards and most
+ modern video standards assume left.
+ * JFIF assumes Rec. 601 Y'CbCr coefficients, while all modern HD processing
+ uses Rec. 709 Y'CbCr coefficients. (Futatabi does not care much about
+ the actual RGB color space; Nageru assumes it is Rec. 709, like for capture
+ cards, but the differences between 601 and 709 here are small. sRGB gamma
+ is assumed throughout, like in JFIF.)
+
+Many players may also be confused by the fact that the resolution can change
+from frame to frame; this is because for original (uninterpolated) frames,
+Futatabi will simply output the received JPEG frame directly to the output
+stream, which can be a different resolution from the interpolated frames.
+
+Also, even though Futatabi exists to make a fixed-framerate stream out of
+something that's not, the output stream can be variable-framerate (VFR) due to
+pauses and frame drops. In particular, when the stream is paused, frames are
+only output about every 100 milliseconds.
+
+Finally, the subtitle track with status information (see :ref:`talkback`) is
+not marked as metadata due to FFmpeg limitations, and as such will show up
+raw in subtitle-enabled players.
+
+
+.. _midi:
+
+Using MIDI controllers
+----------------------
+
+
Monitoring
----------
+.. _talkback:
+
Tally and status talkback
'''''''''''''''''''''''''
+In addition to the verbal communication between the two operators
+(see :ref:`coop`), it is also useful to have automatic communication
+between Nageru and Futatabi. This goes both ways.
+
+First, Futatabi can consume :ref:`Nageru's tally data <tally>` like any camera
+can; give the correct channel URL to *--tally-url* on the Futatabi command line,
+and the color around the live view will show up when you are ready or playing
+(e.g. *--tally-url http://nageru-server.example.org:9096/channels/2*).
+
+Second, Futatabi can export its current status in two ways. The simplest is
+through a REST call; the HTTP server that exposes the stream also exposes
+the endpoint */queue_status* (e.g. http://futatabi-server.example.org:9096/queue_status).
+This contains the same text as is below the live window, ie., how much time
+is queued or left.
+
+The same status is also exported programmatically in the video output from
+Futatabi, as a subtitle track. This allows the Nageru theme not only to display
+it if desired, but even to act automatically to switch to a different channel
+when the playlist is nearing its end. (See :ref:`subtitle-ingest` for information
+on how to digest such information from the Nageru side.)
+
+Each subtitle entry is displayed momentarily before the frame it belongs to,
+and has the following format:
+
+ Futatabi 1.8.2;PLAYING;6.995;0:06.995 left
+
+The semicolon-separated columns are as follows:
+
+ * The first column is the identifier for the Futatabi version in use; if the format
+ should ever diverge between versions, it can serve as a way to distinguish between
+ them if needed. (The version may also change without the format changing.)
+ For now, however, you can ignore it.
+ * The second column is either PLAYING or PAUSED, depending on the current status.
+ * The third column is how many seconds is queued up (PAUSED) or remaining (PLAYING).
+ It is always in C locale, no matter what the user has set (ie., the decimal point
+ will always be a period). Note that this does not take into account the master
+ speed (see :ref:`speed`).
+ * Finally, the fourth column is a human-readable string, the same as is exposed
+ on the /queue_status endpoint above.
+
Prometheus metrics
''''''''''''''''''
projects / nageru-docs / blobdiff