Monitoring
==========
If you have a large Nageru installation, you may want to monitor it remotely,
as opposed to just eyeballing the UI and occasionally checking into the stream
to verify everything is OK. As of version 1.6.1, Nageru supports native
`Prometheus `_ metrics.
This section is not intended to explain Prometheus; for that, see the Prometheus
documentation. You do not need a separate exporter for Nageru, you simply point
Prometheus to the same HTTP port as is used for your stream, and it will fetch
all metrics from the /metrics endpoint.
Nageru's metrics export basically everything you can get from the UI and much more.
Unfortunately, Prometheus is relatively inefficient when it comes to dealing with
histograms, and many of the values Nageru tries to measure (like latency through
the various points in the pipeline) can take on a quite wide range of values.
Thus, most values that would normally be exported as Prometheus histograms are
exported as summaries over one-minute sliding windows instead. This means they
are highly precise, but unfortunately, it means you cannot aggregate them
meaningfully.
Grafana dashboard
-----------------
Nageru ships with an example `Grafana `_ dashboard exported
in JSON format; it does not graph all metrics, but it should cover most of the
interesting parts. Note that there is currently no guarantee of forward or
backward compatibility in metrics, so if Nageru internals change significantly,
some graphs could stop working. Of course, you can change the dashboard as you
see fit; if you don't use e.g. HDMI/SDI output, you will probably want to
remove the panels related to it.
.. _tally:
Tally display
-------------
Not related to Prometheus per se, but yet an important part of monitoring is the
*tally light*, or just tally for short. The tally is a talkback signal from the
mixer to the camera, showing the cameraperson that they are indeed on air (or are
about to get to). Professional studio cameras will often not only have a tally
light for the operator, but also for the subject (to know which camera to look
into). However, most cheaper cameras will have none at all.
Nageru does not have functionality to talk to tally lights directly (there are too
many kinds of interfaces and standards), but it does have functionality to expose
tally *data*. Using this, you can build your own tally light, be in through injecting
the right bits on an SDI cable, having a single-board computer set some GPIO pins
for a LED, or simply show a red or green dot on a mobile phone with a web browser.
Nageru's tally data is automatically exposed over HTTP, just as any other monitoring.
You can get one JSON file with a list of all channels (at “http://yourserver.example.org:9095/channels”),
or you can ask for a specific channel and get just its color in plain text
(at “http://yourserver.example.org:9095/channels/2”, where 2 is the lowest
channel ID, since live and preview don't have tally). The tally color values
are taken automatically from the theme's *channel_color* function, so that they
match the borders around the input in the producer's mixer display—typically
green for on preview and red for live. This also means they can be CSS textual
colors (like “red”), although you can of course make the theme return only
“#ff0000” or similar if this makes your tally application simpler.
Note that the tally endpoints have fully open `CORS `_
headers, so that they can be queried from anywhere; tally data is not sensitive,
and this makes it significantly easier to query them from a web page.