]>
git.sesse.net Git - cubemap/log
Steinar H. Gunderson [Sat, 24 Aug 2013 15:58:02 +0000 (17:58 +0200)]
Include a Munin plugin that was missing in the archive.
Steinar H. Gunderson [Sun, 18 Aug 2013 11:46:22 +0000 (13:46 +0200)]
Fix an issue where rewind-to-HTTP-header would not properly work.
Steinar H. Gunderson [Sun, 18 Aug 2013 11:03:40 +0000 (13:03 +0200)]
Re-run include-what-you-use.
Steinar H. Gunderson [Sun, 18 Aug 2013 09:53:20 +0000 (11:53 +0200)]
Munin plugins should be symlinked, not created directly in /etc.
Steinar H. Gunderson [Sun, 18 Aug 2013 09:52:21 +0000 (11:52 +0200)]
Make the install target install the Munin plugins.
Steinar H. Gunderson [Sun, 18 Aug 2013 08:55:43 +0000 (10:55 +0200)]
Tweak the MutexLock implementation slightly, so as to confuse Coverity less.
Steinar H. Gunderson [Sun, 18 Aug 2013 00:54:54 +0000 (02:54 +0200)]
Compute the checksum on outgoing Metacube packets as well.
Steinar H. Gunderson [Sun, 18 Aug 2013 00:48:13 +0000 (02:48 +0200)]
Use memcpy instead of incurring potential unaligned reads when reading the Metacube header.
This would probably be a portability problem for many other platforms than x86.
Steinar H. Gunderson [Sun, 18 Aug 2013 00:37:16 +0000 (02:37 +0200)]
Switch to a new version of the Metacube protocol.
Compared to the old version, this fixes many small deficiencies:
- The header is smaller; 16 instead of 24 bytes. With TS, Metacube actually
yields noticeable overhead (13% or so), so reducing it by 1/3 is good.
- The sync no longer overlaps with itself; both starting and ending with
the same sounds bad if we should ever drop bytes in the middle.
In fact, it no longer has any repeated characters.
- The header is checksummed, to avoid cases where a corrupted header
could cause us to pull in gigabytes of data. (The data is not.)
vlc-metacube.diff has been updated, although it includes another patch
(part of the WebM patch set) since that's what my VLC works from at the moment.
Steinar H. Gunderson [Sat, 17 Aug 2013 13:09:03 +0000 (15:09 +0200)]
Make the HTTP inputs time out after 30 seconds of no activity.
This is useful for the case where the remote machine is gone,
but TCP doesn't pick it up, and thus you can't easily restart
the streaming.
Steinar H. Gunderson [Sat, 17 Aug 2013 09:01:21 +0000 (11:01 +0200)]
Correct my e-mail address in the man page (cubemap is not written by my Debian hat).
Philipp Kern [Fri, 16 Aug 2013 22:14:24 +0000 (00:14 +0200)]
Add a manual page for cubemap.
Steinar H. Gunderson [Fri, 16 Aug 2013 18:33:57 +0000 (20:33 +0200)]
Store the stream header in the inputs, not just the streams.
This avoids an issue where a stream would get no stream header
when it reused an existing input.
Steinar H. Gunderson [Fri, 16 Aug 2013 15:18:56 +0000 (17:18 +0200)]
When adding new streams that are copies of old streams, copy the HTTP header.
This was already fixed for UDP inputs, but not HTTP inputs. Also, we probably
do not do the right thing here for the stream header.
Steinar H. Gunderson [Fri, 16 Aug 2013 17:11:03 +0000 (19:11 +0200)]
Make an option to properly delete streams.
Steinar H. Gunderson [Fri, 16 Aug 2013 15:43:53 +0000 (17:43 +0200)]
Do not crash on invalid src= URLs.
Steinar H. Gunderson [Fri, 16 Aug 2013 12:14:29 +0000 (14:14 +0200)]
Default PREFIX should be /usr/local, not /usr.
Philipp Kern [Fri, 16 Aug 2013 09:57:17 +0000 (11:57 +0200)]
Makefile: accept LDFLAGS for linking
Steinar H. Gunderson [Fri, 16 Aug 2013 11:39:39 +0000 (13:39 +0200)]
Fix the vertical label of the cubemap_input Munin plugin.
Steinar H. Gunderson [Fri, 16 Aug 2013 09:44:51 +0000 (11:44 +0200)]
Provide an install target in the Makefile.
Based on a patch from Philipp Kern.
Steinar H. Gunderson [Fri, 16 Aug 2013 09:29:54 +0000 (11:29 +0200)]
Added some Munin plugins.
Steinar H. Gunderson [Fri, 16 Aug 2013 00:09:34 +0000 (02:09 +0200)]
Store and log connection time for inputs.
Steinar H. Gunderson [Fri, 16 Aug 2013 00:07:19 +0000 (02:07 +0200)]
Two minor cleanups in the stats code.
Steinar H. Gunderson [Thu, 15 Aug 2013 23:37:43 +0000 (01:37 +0200)]
Add preliminary support for input stream statistics.
Steinar H. Gunderson [Thu, 15 Aug 2013 17:40:08 +0000 (19:40 +0200)]
Fix broken umask (we had the bits inverted).
Steinar H. Gunderson [Wed, 14 Aug 2013 23:02:06 +0000 (01:02 +0200)]
Make the Metacube block size warning a bit less aggressive, as WebM seems to generate blocks of up to 70-80 kB sometimes.
Steinar H. Gunderson [Wed, 14 Aug 2013 21:09:52 +0000 (23:09 +0200)]
Support the new METACUBE_FLAGS_NOT_SUITABLE_FOR_STREAM_START Metacube flag.
This is created by new versions of the VLC Metacube patches;
it is designed for the needs of WebM, where browsers refuse to accept a
stream that does not start with a keyframe. Thus, we can have blocks that
are completely valid (and we wouldn't want to start joining blocks, as the
GOPs can become very large), but that we do not want to start a stream with.
For this, we introduce a new state called WAITING_FOR_KEYFRAME that all
new clients (except those fetching from the start of the backlog) go through;
they stay there until the next keyframe-marked block (anything that's not
marked METACUBE_FLAGS_NOT_SUITABLE_FOR_STREAM_START, so old Metacube streams
are still valid) comes along, at which point they go into the regular
SENDING_DATA.
The new version of the VLC Metacube patch is not included yet.
Steinar H. Gunderson [Wed, 14 Aug 2013 19:34:24 +0000 (21:34 +0200)]
When warning about unusually long Metacube blocks, use decimal instead of hex.
Steinar H. Gunderson [Tue, 13 Aug 2013 14:08:01 +0000 (16:08 +0200)]
When reading the HTTP input, skip the \r\n\r\n. Fixes the problem of 4 non-Metacube bytes being skipped.
Steinar H. Gunderson [Tue, 13 Aug 2013 09:42:41 +0000 (11:42 +0200)]
Warn on large Metacube blocks.
Steinar H. Gunderson [Tue, 13 Aug 2013 09:13:38 +0000 (11:13 +0200)]
Add an assert, to hopefully catch a bug.
Steinar H. Gunderson [Mon, 12 Aug 2013 21:37:17 +0000 (23:37 +0200)]
Set num_servers to 1 in the sample config, as that is the more typical situation.
Steinar H. Gunderson [Thu, 6 Jun 2013 00:21:46 +0000 (02:21 +0200)]
(Hopefully) fix an assert failure in httpinput.cpp.
If we get the Metacube sync marker but not the right of the header,
we can set has_metacube_header, and then return. If the connection
drops right there, the flag won't be correctly reset, and if the
next connection starts with something that's not a Metacube block,
we will try to read junk data as Metacube.
Thankfully, the assert caught this case.
Steinar H. Gunderson [Mon, 20 May 2013 13:47:41 +0000 (15:47 +0200)]
Fix a possible corruption/crash when name lookups fail, due to incorrect error checking of the getaddrinfo() result.
Steinar H. Gunderson [Wed, 15 May 2013 20:26:31 +0000 (22:26 +0200)]
Fix a possible infinite loop in HTTPInput.
Steinar H. Gunderson [Wed, 15 May 2013 20:24:54 +0000 (22:24 +0200)]
Fix a potential double-free.
Steinar H. Gunderson [Wed, 8 May 2013 18:45:46 +0000 (20:45 +0200)]
Merge branch 'master' of /srv/git.sesse.net/www/cubemap
Steinar H. Gunderson [Wed, 8 May 2013 18:45:27 +0000 (20:45 +0200)]
Unbreak access logging.
We forgot to wake up the logger thread.
Steinar H. Gunderson [Sun, 28 Apr 2013 16:48:24 +0000 (18:48 +0200)]
Add a hack for starting at the _start_ of backlog. Useful for fetching samples of streams crashing VLC after-the-back.
Steinar H. Gunderson [Sat, 27 Apr 2013 11:54:52 +0000 (13:54 +0200)]
Never send more than IOV_MAX (1024) iovecs in pwritev.
This fixes issues with “pwritev: Invalid argument” showing up
under rare circumstances. Apparently the glibc wrapper workaround
described in writev(2) does not work, or does not work for pwritev().
Steinar H. Gunderson [Wed, 24 Apr 2013 22:18:53 +0000 (00:18 +0200)]
Fix a minor error message.
Steinar H. Gunderson [Mon, 22 Apr 2013 21:24:59 +0000 (23:24 +0200)]
Enable TCP_CORK if available.
Steinar H. Gunderson [Sun, 21 Apr 2013 22:52:44 +0000 (00:52 +0200)]
Support UDP packets larger than 4 kB.
Steinar H. Gunderson [Sun, 21 Apr 2013 19:48:00 +0000 (21:48 +0200)]
Timestamp the error log.
Steinar H. Gunderson [Sun, 21 Apr 2013 19:12:39 +0000 (21:12 +0200)]
Catch up on any lost data before serializing, so increasing the backlog will not inject zero bytes into the data of (too) backlogged clients if we increase the backlog size.
Steinar H. Gunderson [Sun, 21 Apr 2013 14:54:52 +0000 (16:54 +0200)]
Bump version number to 0.2; it's about time.
Steinar H. Gunderson [Sun, 21 Apr 2013 12:00:27 +0000 (14:00 +0200)]
Small error message fix.
Steinar H. Gunderson [Sun, 21 Apr 2013 11:57:00 +0000 (13:57 +0200)]
Run include-what-you-use.
Steinar H. Gunderson [Sun, 21 Apr 2013 11:56:03 +0000 (13:56 +0200)]
Merge branch 'master' of /srv/git.sesse.net/www/cubemap
Steinar H. Gunderson [Sun, 21 Apr 2013 11:55:23 +0000 (13:55 +0200)]
Add support for UDP outputs.
Steinar H. Gunderson [Sun, 21 Apr 2013 10:48:27 +0000 (12:48 +0200)]
Fix a Valgrind hit that could cause an undefined set of signals to be blocked.
Steinar H. Gunderson [Sun, 21 Apr 2013 10:31:01 +0000 (12:31 +0200)]
Minor cleanup.
Steinar H. Gunderson [Sun, 21 Apr 2013 00:26:45 +0000 (02:26 +0200)]
Merge branch 'master' of /srv/git.sesse.net/www/cubemap
Steinar H. Gunderson [Sun, 21 Apr 2013 00:26:35 +0000 (02:26 +0200)]
Fix a (harmless) close() warning that would come if we deleted a HTTP input that was not connected.
Steinar H. Gunderson [Sat, 20 Apr 2013 23:55:57 +0000 (01:55 +0200)]
Run include-what-you-use.
Steinar H. Gunderson [Sat, 20 Apr 2013 23:51:52 +0000 (01:51 +0200)]
Refer to streams internally mostly by an index, not the stream_id.
Inputs no longer know about stream_id, only the stream_index.
This is faster (we used maybe 5% of our time in find_stream),
and will make it easier to add non-HTTP outputs in the future.
Within (HTTP) clients, stream_id also no longer exists, but is replaced
by "url", which really works exactly the same way. url is mostly used during
re-exec, however, so it's less important than it used to be.
Steinar H. Gunderson [Sat, 20 Apr 2013 14:41:18 +0000 (16:41 +0200)]
Fix merge snafu in last commit. (Oops.)
Steinar H. Gunderson [Sat, 20 Apr 2013 14:39:20 +0000 (16:39 +0200)]
Small optimization to Metacube output.
Steinar H. Gunderson [Sat, 20 Apr 2013 14:30:16 +0000 (16:30 +0200)]
Move iovecs around instead of having single data buffers. Hopefully a tad more efficient, although it does mean somewhat more malloc-ing.
Steinar H. Gunderson [Sat, 20 Apr 2013 13:56:57 +0000 (15:56 +0200)]
Add Metacube headers in add_data_deferred(), not add_data().
This fixes a problem where Metacube blocks are not deterministically added
for each add_data() call (because they would only be added for each call to
process_queued_data()), causing the different servers to go out of sync.
Also moved most of this logic to Stream, where it seems more appropriate
(even though the mutexes are a bit odd now).
Steinar H. Gunderson [Sat, 20 Apr 2013 12:25:24 +0000 (14:25 +0200)]
When unwrapping in set_backlog_size, do not add a spurious Metacube header.
Steinar H. Gunderson [Sat, 20 Apr 2013 12:12:41 +0000 (14:12 +0200)]
Revert "Fix a bug where Metacube streams would become inconsistent between different servers, since the first one did not get the right encoding set."
Doesn't actually fix the issue.
This reverts commit
5605c88a27fcf64ea8f6fd6cbb52dada73856ee0 .
Steinar H. Gunderson [Sat, 20 Apr 2013 12:00:02 +0000 (14:00 +0200)]
Fix a bug where Metacube streams would become inconsistent between different servers, since the first one did not get the right encoding set.
Steinar H. Gunderson [Sat, 20 Apr 2013 00:19:22 +0000 (02:19 +0200)]
Reinstate the new signal handling; revert the revert.
The fix for high CPU usage was not doing wakeup() all the time for
new data; not only can seemingly sending the signal be expensive,
but we sent it while still holding the mutex the awakening thread
needed, so it was extra-bad.
We're seemingly down to a reasonable amount of CPU usage again now,
but still with good non-racing semantics.
Steinar H. Gunderson [Fri, 19 Apr 2013 23:21:15 +0000 (01:21 +0200)]
Revert "Rewrite the entire internal signal handling/wakeup."
Seemingly this had very bad effects on CPU usage. Will need to
investigate later.
This reverts commit
3fd8650ccf3da3960a946d8ac9abc305aec399ce .
Steinar H. Gunderson [Fri, 19 Apr 2013 22:59:54 +0000 (00:59 +0200)]
Fix broken linking.
Steinar H. Gunderson [Fri, 19 Apr 2013 21:44:23 +0000 (23:44 +0200)]
Compile and link with -pthread, not -lpthread.
Steinar H. Gunderson [Fri, 19 Apr 2013 21:39:23 +0000 (23:39 +0200)]
Rewrite the entire internal signal handling/wakeup.
We now solve things in a much less racy way, sending SIGUSR1 internally,
which is blocked except in poll/epoll. This should be more stable,
require no more fixed-length wakeups (no more wakeup every 20 ms in
each server), and upset Valgrind less.
However, it's also complex to get right, so it might introduce new bugs.
Steinar H. Gunderson [Fri, 19 Apr 2013 19:08:08 +0000 (21:08 +0200)]
Remove unneeded default Client constructor.
Steinar H. Gunderson [Fri, 19 Apr 2013 18:48:35 +0000 (20:48 +0200)]
Fix 100% CPU usage when there is no access_log.
Steinar H. Gunderson [Fri, 19 Apr 2013 17:57:15 +0000 (19:57 +0200)]
Do not crash on re-exec if we do not have an access log.
Steinar H. Gunderson [Fri, 19 Apr 2013 17:40:38 +0000 (19:40 +0200)]
Fix some issues with the last stats.cpp fix.
Steinar H. Gunderson [Fri, 19 Apr 2013 17:28:22 +0000 (19:28 +0200)]
Fix more missing log_perror() calls.
There's nothing we can do about them, but it's good to report.
Found by Coverity Scan.
Steinar H. Gunderson [Fri, 19 Apr 2013 17:25:59 +0000 (19:25 +0200)]
Add a missing log_perror() on close() failure (cosmetic issue).
Found by Coverity Scan.
Steinar H. Gunderson [Fri, 19 Apr 2013 17:23:24 +0000 (19:23 +0200)]
Set umask security before calling mkstemp().
Found by Coverity Scan.
Steinar H. Gunderson [Fri, 19 Apr 2013 17:13:45 +0000 (19:13 +0200)]
If make_tempfile() returns -1 in set_backlog_size, exit properly.
Found by Coverity Scan.
Steinar H. Gunderson [Thu, 18 Apr 2013 23:39:53 +0000 (01:39 +0200)]
New run of include-what-you-use.
Steinar H. Gunderson [Thu, 18 Apr 2013 23:38:36 +0000 (01:38 +0200)]
Set Connection: close in outgoing HTTP headers.
Steinar H. Gunderson [Thu, 18 Apr 2013 23:33:33 +0000 (01:33 +0200)]
When logging losses, log loss percentage, too.
Steinar H. Gunderson [Thu, 18 Apr 2013 23:27:24 +0000 (01:27 +0200)]
Show IPv4 clients by their IPv4 address instead of with a v4-mapped address.
Steinar H. Gunderson [Thu, 18 Apr 2013 22:41:19 +0000 (00:41 +0200)]
Send backlog file descriptors around instead of going through the protobuf. Much faster restart with big backlog sizes, and does not hit the 64MB protobuf limit.
Steinar H. Gunderson [Thu, 18 Apr 2013 19:28:10 +0000 (21:28 +0200)]
Fix an issue where new UDP streams would be without HTTP headers.
Steinar H. Gunderson [Wed, 17 Apr 2013 23:10:14 +0000 (01:10 +0200)]
Do not bother making a METACUBE_FLAGS_HEADER block if it is empty.
Steinar H. Gunderson [Wed, 17 Apr 2013 23:07:41 +0000 (01:07 +0200)]
When closing unused inputs, remember to take down their sockets.
Steinar H. Gunderson [Wed, 17 Apr 2013 23:02:28 +0000 (01:02 +0200)]
Support Metacube _output_. Required splitting HTTP headers from stream headers, which was a good thing anyway, but a bit painful in upgrades.
Steinar H. Gunderson [Wed, 17 Apr 2013 21:36:46 +0000 (23:36 +0200)]
Deduplicate inputs.
Steinar H. Gunderson [Wed, 17 Apr 2013 18:14:59 +0000 (20:14 +0200)]
Do not segfault on unknown options.
Steinar H. Gunderson [Tue, 16 Apr 2013 20:39:46 +0000 (22:39 +0200)]
Fix a leak in the stats thread.
Steinar H. Gunderson [Tue, 16 Apr 2013 20:38:20 +0000 (22:38 +0200)]
Fix another minor leak.
Steinar H. Gunderson [Tue, 16 Apr 2013 20:38:09 +0000 (22:38 +0200)]
Stop leaking the mark pools.
Steinar H. Gunderson [Tue, 16 Apr 2013 20:35:06 +0000 (22:35 +0200)]
Fix a memory leak in Server (streams were leaked).
Steinar H. Gunderson [Tue, 16 Apr 2013 20:33:11 +0000 (22:33 +0200)]
Fix a small memory leak in HTTPInput.
Steinar H. Gunderson [Tue, 16 Apr 2013 20:30:47 +0000 (22:30 +0200)]
Fix a Valgrind hit.
Steinar H. Gunderson [Tue, 16 Apr 2013 20:30:30 +0000 (22:30 +0200)]
Fix a Valgrind hit.
Steinar H. Gunderson [Tue, 16 Apr 2013 20:29:39 +0000 (22:29 +0200)]
Log all finished accesses to an access log.
Steinar H. Gunderson [Tue, 16 Apr 2013 19:47:56 +0000 (21:47 +0200)]
Less confusing error message when realpath() fails.
Steinar H. Gunderson [Tue, 16 Apr 2013 18:08:06 +0000 (20:08 +0200)]
Clearer warnings in the cubemap.config.sample file about daemonizing and paths.
Steinar H. Gunderson [Tue, 16 Apr 2013 18:06:51 +0000 (20:06 +0200)]
Run --config-test with the canonicalized paths.
Steinar H. Gunderson [Tue, 16 Apr 2013 18:01:26 +0000 (20:01 +0200)]
Support daemonizing.
Steinar H. Gunderson [Tue, 16 Apr 2013 17:57:51 +0000 (19:57 +0200)]
Canonicalize paths, so that we do not need to worry about cwd on re-exec.