]>
git.sesse.net Git - cubemap/log
Steinar H. Gunderson [Sat, 27 Apr 2013 11:54:52 +0000 (13:54 +0200)]
Never send more than IOV_MAX (1024) iovecs in pwritev.
This fixes issues with “pwritev: Invalid argument” showing up
under rare circumstances. Apparently the glibc wrapper workaround
described in writev(2) does not work, or does not work for pwritev().
Steinar H. Gunderson [Wed, 24 Apr 2013 22:18:53 +0000 (00:18 +0200)]
Fix a minor error message.
Steinar H. Gunderson [Mon, 22 Apr 2013 21:24:59 +0000 (23:24 +0200)]
Enable TCP_CORK if available.
Steinar H. Gunderson [Sun, 21 Apr 2013 22:52:44 +0000 (00:52 +0200)]
Support UDP packets larger than 4 kB.
Steinar H. Gunderson [Sun, 21 Apr 2013 19:48:00 +0000 (21:48 +0200)]
Timestamp the error log.
Steinar H. Gunderson [Sun, 21 Apr 2013 19:12:39 +0000 (21:12 +0200)]
Catch up on any lost data before serializing, so increasing the backlog will not inject zero bytes into the data of (too) backlogged clients if we increase the backlog size.
Steinar H. Gunderson [Sun, 21 Apr 2013 14:54:52 +0000 (16:54 +0200)]
Bump version number to 0.2; it's about time.
Steinar H. Gunderson [Sun, 21 Apr 2013 12:00:27 +0000 (14:00 +0200)]
Small error message fix.
Steinar H. Gunderson [Sun, 21 Apr 2013 11:57:00 +0000 (13:57 +0200)]
Run include-what-you-use.
Steinar H. Gunderson [Sun, 21 Apr 2013 11:56:03 +0000 (13:56 +0200)]
Merge branch 'master' of /srv/git.sesse.net/www/cubemap
Steinar H. Gunderson [Sun, 21 Apr 2013 11:55:23 +0000 (13:55 +0200)]
Add support for UDP outputs.
Steinar H. Gunderson [Sun, 21 Apr 2013 10:48:27 +0000 (12:48 +0200)]
Fix a Valgrind hit that could cause an undefined set of signals to be blocked.
Steinar H. Gunderson [Sun, 21 Apr 2013 10:31:01 +0000 (12:31 +0200)]
Minor cleanup.
Steinar H. Gunderson [Sun, 21 Apr 2013 00:26:45 +0000 (02:26 +0200)]
Merge branch 'master' of /srv/git.sesse.net/www/cubemap
Steinar H. Gunderson [Sun, 21 Apr 2013 00:26:35 +0000 (02:26 +0200)]
Fix a (harmless) close() warning that would come if we deleted a HTTP input that was not connected.
Steinar H. Gunderson [Sat, 20 Apr 2013 23:55:57 +0000 (01:55 +0200)]
Run include-what-you-use.
Steinar H. Gunderson [Sat, 20 Apr 2013 23:51:52 +0000 (01:51 +0200)]
Refer to streams internally mostly by an index, not the stream_id.
Inputs no longer know about stream_id, only the stream_index.
This is faster (we used maybe 5% of our time in find_stream),
and will make it easier to add non-HTTP outputs in the future.
Within (HTTP) clients, stream_id also no longer exists, but is replaced
by "url", which really works exactly the same way. url is mostly used during
re-exec, however, so it's less important than it used to be.
Steinar H. Gunderson [Sat, 20 Apr 2013 14:41:18 +0000 (16:41 +0200)]
Fix merge snafu in last commit. (Oops.)
Steinar H. Gunderson [Sat, 20 Apr 2013 14:39:20 +0000 (16:39 +0200)]
Small optimization to Metacube output.
Steinar H. Gunderson [Sat, 20 Apr 2013 14:30:16 +0000 (16:30 +0200)]
Move iovecs around instead of having single data buffers. Hopefully a tad more efficient, although it does mean somewhat more malloc-ing.
Steinar H. Gunderson [Sat, 20 Apr 2013 13:56:57 +0000 (15:56 +0200)]
Add Metacube headers in add_data_deferred(), not add_data().
This fixes a problem where Metacube blocks are not deterministically added
for each add_data() call (because they would only be added for each call to
process_queued_data()), causing the different servers to go out of sync.
Also moved most of this logic to Stream, where it seems more appropriate
(even though the mutexes are a bit odd now).
Steinar H. Gunderson [Sat, 20 Apr 2013 12:25:24 +0000 (14:25 +0200)]
When unwrapping in set_backlog_size, do not add a spurious Metacube header.
Steinar H. Gunderson [Sat, 20 Apr 2013 12:12:41 +0000 (14:12 +0200)]
Revert "Fix a bug where Metacube streams would become inconsistent between different servers, since the first one did not get the right encoding set."
Doesn't actually fix the issue.
This reverts commit
5605c88a27fcf64ea8f6fd6cbb52dada73856ee0 .
Steinar H. Gunderson [Sat, 20 Apr 2013 12:00:02 +0000 (14:00 +0200)]
Fix a bug where Metacube streams would become inconsistent between different servers, since the first one did not get the right encoding set.
Steinar H. Gunderson [Sat, 20 Apr 2013 00:19:22 +0000 (02:19 +0200)]
Reinstate the new signal handling; revert the revert.
The fix for high CPU usage was not doing wakeup() all the time for
new data; not only can seemingly sending the signal be expensive,
but we sent it while still holding the mutex the awakening thread
needed, so it was extra-bad.
We're seemingly down to a reasonable amount of CPU usage again now,
but still with good non-racing semantics.
Steinar H. Gunderson [Fri, 19 Apr 2013 23:21:15 +0000 (01:21 +0200)]
Revert "Rewrite the entire internal signal handling/wakeup."
Seemingly this had very bad effects on CPU usage. Will need to
investigate later.
This reverts commit
3fd8650ccf3da3960a946d8ac9abc305aec399ce .
Steinar H. Gunderson [Fri, 19 Apr 2013 22:59:54 +0000 (00:59 +0200)]
Fix broken linking.
Steinar H. Gunderson [Fri, 19 Apr 2013 21:44:23 +0000 (23:44 +0200)]
Compile and link with -pthread, not -lpthread.
Steinar H. Gunderson [Fri, 19 Apr 2013 21:39:23 +0000 (23:39 +0200)]
Rewrite the entire internal signal handling/wakeup.
We now solve things in a much less racy way, sending SIGUSR1 internally,
which is blocked except in poll/epoll. This should be more stable,
require no more fixed-length wakeups (no more wakeup every 20 ms in
each server), and upset Valgrind less.
However, it's also complex to get right, so it might introduce new bugs.
Steinar H. Gunderson [Fri, 19 Apr 2013 19:08:08 +0000 (21:08 +0200)]
Remove unneeded default Client constructor.
Steinar H. Gunderson [Fri, 19 Apr 2013 18:48:35 +0000 (20:48 +0200)]
Fix 100% CPU usage when there is no access_log.
Steinar H. Gunderson [Fri, 19 Apr 2013 17:57:15 +0000 (19:57 +0200)]
Do not crash on re-exec if we do not have an access log.
Steinar H. Gunderson [Fri, 19 Apr 2013 17:40:38 +0000 (19:40 +0200)]
Fix some issues with the last stats.cpp fix.
Steinar H. Gunderson [Fri, 19 Apr 2013 17:28:22 +0000 (19:28 +0200)]
Fix more missing log_perror() calls.
There's nothing we can do about them, but it's good to report.
Found by Coverity Scan.
Steinar H. Gunderson [Fri, 19 Apr 2013 17:25:59 +0000 (19:25 +0200)]
Add a missing log_perror() on close() failure (cosmetic issue).
Found by Coverity Scan.
Steinar H. Gunderson [Fri, 19 Apr 2013 17:23:24 +0000 (19:23 +0200)]
Set umask security before calling mkstemp().
Found by Coverity Scan.
Steinar H. Gunderson [Fri, 19 Apr 2013 17:13:45 +0000 (19:13 +0200)]
If make_tempfile() returns -1 in set_backlog_size, exit properly.
Found by Coverity Scan.
Steinar H. Gunderson [Thu, 18 Apr 2013 23:39:53 +0000 (01:39 +0200)]
New run of include-what-you-use.
Steinar H. Gunderson [Thu, 18 Apr 2013 23:38:36 +0000 (01:38 +0200)]
Set Connection: close in outgoing HTTP headers.
Steinar H. Gunderson [Thu, 18 Apr 2013 23:33:33 +0000 (01:33 +0200)]
When logging losses, log loss percentage, too.
Steinar H. Gunderson [Thu, 18 Apr 2013 23:27:24 +0000 (01:27 +0200)]
Show IPv4 clients by their IPv4 address instead of with a v4-mapped address.
Steinar H. Gunderson [Thu, 18 Apr 2013 22:41:19 +0000 (00:41 +0200)]
Send backlog file descriptors around instead of going through the protobuf. Much faster restart with big backlog sizes, and does not hit the 64MB protobuf limit.
Steinar H. Gunderson [Thu, 18 Apr 2013 19:28:10 +0000 (21:28 +0200)]
Fix an issue where new UDP streams would be without HTTP headers.
Steinar H. Gunderson [Wed, 17 Apr 2013 23:10:14 +0000 (01:10 +0200)]
Do not bother making a METACUBE_FLAGS_HEADER block if it is empty.
Steinar H. Gunderson [Wed, 17 Apr 2013 23:07:41 +0000 (01:07 +0200)]
When closing unused inputs, remember to take down their sockets.
Steinar H. Gunderson [Wed, 17 Apr 2013 23:02:28 +0000 (01:02 +0200)]
Support Metacube _output_. Required splitting HTTP headers from stream headers, which was a good thing anyway, but a bit painful in upgrades.
Steinar H. Gunderson [Wed, 17 Apr 2013 21:36:46 +0000 (23:36 +0200)]
Deduplicate inputs.
Steinar H. Gunderson [Wed, 17 Apr 2013 18:14:59 +0000 (20:14 +0200)]
Do not segfault on unknown options.
Steinar H. Gunderson [Tue, 16 Apr 2013 20:39:46 +0000 (22:39 +0200)]
Fix a leak in the stats thread.
Steinar H. Gunderson [Tue, 16 Apr 2013 20:38:20 +0000 (22:38 +0200)]
Fix another minor leak.
Steinar H. Gunderson [Tue, 16 Apr 2013 20:38:09 +0000 (22:38 +0200)]
Stop leaking the mark pools.
Steinar H. Gunderson [Tue, 16 Apr 2013 20:35:06 +0000 (22:35 +0200)]
Fix a memory leak in Server (streams were leaked).
Steinar H. Gunderson [Tue, 16 Apr 2013 20:33:11 +0000 (22:33 +0200)]
Fix a small memory leak in HTTPInput.
Steinar H. Gunderson [Tue, 16 Apr 2013 20:30:47 +0000 (22:30 +0200)]
Fix a Valgrind hit.
Steinar H. Gunderson [Tue, 16 Apr 2013 20:30:30 +0000 (22:30 +0200)]
Fix a Valgrind hit.
Steinar H. Gunderson [Tue, 16 Apr 2013 20:29:39 +0000 (22:29 +0200)]
Log all finished accesses to an access log.
Steinar H. Gunderson [Tue, 16 Apr 2013 19:47:56 +0000 (21:47 +0200)]
Less confusing error message when realpath() fails.
Steinar H. Gunderson [Tue, 16 Apr 2013 18:08:06 +0000 (20:08 +0200)]
Clearer warnings in the cubemap.config.sample file about daemonizing and paths.
Steinar H. Gunderson [Tue, 16 Apr 2013 18:06:51 +0000 (20:06 +0200)]
Run --config-test with the canonicalized paths.
Steinar H. Gunderson [Tue, 16 Apr 2013 18:01:26 +0000 (20:01 +0200)]
Support daemonizing.
Steinar H. Gunderson [Tue, 16 Apr 2013 17:57:51 +0000 (19:57 +0200)]
Canonicalize paths, so that we do not need to worry about cwd on re-exec.
Steinar H. Gunderson [Mon, 15 Apr 2013 23:31:24 +0000 (01:31 +0200)]
Shut down cleanly on SIGINT (Ctrl-C), for easier Valgrind runs.
Steinar H. Gunderson [Mon, 15 Apr 2013 23:15:55 +0000 (01:15 +0200)]
Drop NO_LEVEL; it does not look very pretty now.
Steinar H. Gunderson [Mon, 15 Apr 2013 23:09:20 +0000 (01:09 +0200)]
Log when we have a successful HTTPInput request.
Steinar H. Gunderson [Mon, 15 Apr 2013 23:08:01 +0000 (01:08 +0200)]
Identify UDPInput error messages by the stream, too.
Steinar H. Gunderson [Mon, 15 Apr 2013 23:07:37 +0000 (01:07 +0200)]
Identify HTTPInput error messages by the stream.
Steinar H. Gunderson [Mon, 15 Apr 2013 23:04:53 +0000 (01:04 +0200)]
Make things line up a bit better.
Steinar H. Gunderson [Mon, 15 Apr 2013 22:41:10 +0000 (00:41 +0200)]
Merge branch 'master' of /srv/git.sesse.net/www/cubemap
Steinar H. Gunderson [Mon, 15 Apr 2013 22:41:04 +0000 (00:41 +0200)]
Correct use of strerror_r(); we always need to use the return value.
Steinar H. Gunderson [Mon, 15 Apr 2013 22:39:36 +0000 (00:39 +0200)]
Merge branch 'master' of /srv/git.sesse.net/www/cubemap
Steinar H. Gunderson [Mon, 15 Apr 2013 22:39:17 +0000 (00:39 +0200)]
Store socket and fwmark in the stats file.
Steinar H. Gunderson [Mon, 15 Apr 2013 22:37:31 +0000 (00:37 +0200)]
Log IP address instead of file descriptor.
Steinar H. Gunderson [Mon, 15 Apr 2013 22:33:34 +0000 (00:33 +0200)]
Merge branch 'master' of /srv/git.sesse.net/www/cubemap
Steinar H. Gunderson [Mon, 15 Apr 2013 22:32:52 +0000 (00:32 +0200)]
Replace all perror() calls with our own log calls.
Steinar H. Gunderson [Mon, 15 Apr 2013 21:27:58 +0000 (23:27 +0200)]
Merge branch 'master' of /srv/git.sesse.net/www/cubemap
Steinar H. Gunderson [Mon, 15 Apr 2013 21:26:56 +0000 (23:26 +0200)]
Document the new logging options in the example config.
Steinar H. Gunderson [Mon, 15 Apr 2013 21:24:38 +0000 (23:24 +0200)]
Implement much better logging, with support for both files and syslog.
Steinar H. Gunderson [Mon, 15 Apr 2013 10:20:37 +0000 (12:20 +0200)]
Merge branch 'master' of /srv/git.sesse.net/www/cubemap
Steinar H. Gunderson [Mon, 15 Apr 2013 08:37:18 +0000 (10:37 +0200)]
Fix a signed/unsigned warning.
Steinar H. Gunderson [Mon, 15 Apr 2013 08:36:33 +0000 (10:36 +0200)]
Store the client pointer directly in the epoll structure, instead of doing a map lookup every time. Saves some CPU.
Steinar H. Gunderson [Sun, 14 Apr 2013 21:41:22 +0000 (23:41 +0200)]
Small microoptimization in do_work().
Steinar H. Gunderson [Sun, 14 Apr 2013 21:24:33 +0000 (23:24 +0200)]
We now have benchmark results.
Steinar H. Gunderson [Sun, 14 Apr 2013 18:54:25 +0000 (20:54 +0200)]
Ignore SIGPIPE, so we do not die when a client shuts down at the wrong time.
Steinar H. Gunderson [Sun, 14 Apr 2013 18:21:25 +0000 (20:21 +0200)]
Make read_tempfile() much faster, and make it close the file also on error.
Steinar H. Gunderson [Sun, 14 Apr 2013 18:10:49 +0000 (20:10 +0200)]
Do not start the servers until all the deserialized clients have been put back. Fixes an assert failure on load, and makes for faster loads.
Steinar H. Gunderson [Sun, 14 Apr 2013 17:53:48 +0000 (19:53 +0200)]
Signal thread stop through a pipe; fixes issues where the statistics thread would wait one round on shutdown.
Steinar H. Gunderson [Sun, 14 Apr 2013 16:49:05 +0000 (18:49 +0200)]
Remove an include IWYU should not have added.
Steinar H. Gunderson [Sun, 14 Apr 2013 16:48:02 +0000 (18:48 +0200)]
Move SIGHUP handling as soon as possible, so that repeated SIGHUPing is less likely to kill us.
Steinar H. Gunderson [Sun, 14 Apr 2013 16:13:19 +0000 (18:13 +0200)]
Log bytes sent and lost better.
Steinar H. Gunderson [Sun, 14 Apr 2013 16:13:08 +0000 (18:13 +0200)]
Fix a crash on re-exec if a client was not in SENDING_DATA.
Steinar H. Gunderson [Sun, 14 Apr 2013 00:30:47 +0000 (02:30 +0200)]
Rename bytes_sent to stream_pos, because that is what it really is.
Steinar H. Gunderson [Sat, 13 Apr 2013 21:03:49 +0000 (23:03 +0200)]
Make more usable documentation.
Steinar H. Gunderson [Sat, 13 Apr 2013 20:53:23 +0000 (22:53 +0200)]
Merge branch 'master' of ssh://git.sesse.net/srv/git.sesse.net/www/cubemap
Steinar H. Gunderson [Sat, 13 Apr 2013 20:53:02 +0000 (22:53 +0200)]
When HTTPInput disconnects, also clear the header.
Steinar H. Gunderson [Sat, 13 Apr 2013 20:48:38 +0000 (22:48 +0200)]
Another include-what-you-use pass.
Steinar H. Gunderson [Sat, 13 Apr 2013 20:47:24 +0000 (22:47 +0200)]
Make backlog_size changeable across HUPs.
Steinar H. Gunderson [Sat, 13 Apr 2013 20:35:45 +0000 (22:35 +0200)]
Move Server:add_data() into Stream, where it more logically belongs.
Steinar H. Gunderson [Sat, 13 Apr 2013 20:13:56 +0000 (22:13 +0200)]
Run include-what-you-use.
Steinar H. Gunderson [Sat, 13 Apr 2013 20:12:53 +0000 (22:12 +0200)]
Fix incorrect struct/class in forward declares (found by Clang).
Steinar H. Gunderson [Sat, 13 Apr 2013 20:02:18 +0000 (22:02 +0200)]
Move Client and Stream into their own files.