cubemap
7 years agoMerge branch 'master' of /srv/git.sesse.net/www/cubemap
Steinar H. Gunderson [Mon, 15 Apr 2013 22:39:36 +0000 (00:39 +0200)]
Merge branch 'master' of /srv/git.sesse.net/www/cubemap

7 years agoStore socket and fwmark in the stats file.
Steinar H. Gunderson [Mon, 15 Apr 2013 22:39:17 +0000 (00:39 +0200)]
Store socket and fwmark in the stats file.

7 years agoLog IP address instead of file descriptor.
Steinar H. Gunderson [Mon, 15 Apr 2013 22:37:31 +0000 (00:37 +0200)]
Log IP address instead of file descriptor.

7 years agoMerge branch 'master' of /srv/git.sesse.net/www/cubemap
Steinar H. Gunderson [Mon, 15 Apr 2013 22:33:34 +0000 (00:33 +0200)]
Merge branch 'master' of /srv/git.sesse.net/www/cubemap

7 years agoReplace all perror() calls with our own log calls.
Steinar H. Gunderson [Mon, 15 Apr 2013 22:32:52 +0000 (00:32 +0200)]
Replace all perror() calls with our own log calls.

7 years agoMerge branch 'master' of /srv/git.sesse.net/www/cubemap
Steinar H. Gunderson [Mon, 15 Apr 2013 21:27:58 +0000 (23:27 +0200)]
Merge branch 'master' of /srv/git.sesse.net/www/cubemap

7 years agoDocument the new logging options in the example config.
Steinar H. Gunderson [Mon, 15 Apr 2013 21:26:56 +0000 (23:26 +0200)]
Document the new logging options in the example config.

7 years agoImplement much better logging, with support for both files and syslog.
Steinar H. Gunderson [Mon, 15 Apr 2013 21:24:38 +0000 (23:24 +0200)]
Implement much better logging, with support for both files and syslog.

7 years agoMerge branch 'master' of /srv/git.sesse.net/www/cubemap
Steinar H. Gunderson [Mon, 15 Apr 2013 10:20:37 +0000 (12:20 +0200)]
Merge branch 'master' of /srv/git.sesse.net/www/cubemap

7 years agoFix a signed/unsigned warning.
Steinar H. Gunderson [Mon, 15 Apr 2013 08:37:18 +0000 (10:37 +0200)]
Fix a signed/unsigned warning.

7 years agoStore the client pointer directly in the epoll structure, instead of doing a map...
Steinar H. Gunderson [Mon, 15 Apr 2013 08:36:33 +0000 (10:36 +0200)]
Store the client pointer directly in the epoll structure, instead of doing a map lookup every time. Saves some CPU.

7 years agoSmall microoptimization in do_work().
Steinar H. Gunderson [Sun, 14 Apr 2013 21:41:22 +0000 (23:41 +0200)]
Small microoptimization in do_work().

7 years agoWe now have benchmark results.
Steinar H. Gunderson [Sun, 14 Apr 2013 21:24:33 +0000 (23:24 +0200)]
We now have benchmark results.

7 years agoIgnore SIGPIPE, so we do not die when a client shuts down at the wrong time.
Steinar H. Gunderson [Sun, 14 Apr 2013 18:54:25 +0000 (20:54 +0200)]
Ignore SIGPIPE, so we do not die when a client shuts down at the wrong time.

7 years agoMake read_tempfile() much faster, and make it close the file also on error.
Steinar H. Gunderson [Sun, 14 Apr 2013 18:21:25 +0000 (20:21 +0200)]
Make read_tempfile() much faster, and make it close the file also on error.

7 years agoDo not start the servers until all the deserialized clients have been put back. Fixes...
Steinar H. Gunderson [Sun, 14 Apr 2013 18:10:49 +0000 (20:10 +0200)]
Do not start the servers until all the deserialized clients have been put back. Fixes an assert failure on load, and makes for faster loads.

7 years agoSignal thread stop through a pipe; fixes issues where the statistics thread would...
Steinar H. Gunderson [Sun, 14 Apr 2013 17:53:48 +0000 (19:53 +0200)]
Signal thread stop through a pipe; fixes issues where the statistics thread would wait one round on shutdown.

7 years agoRemove an include IWYU should not have added.
Steinar H. Gunderson [Sun, 14 Apr 2013 16:49:05 +0000 (18:49 +0200)]
Remove an include IWYU should not have added.

7 years agoMove SIGHUP handling as soon as possible, so that repeated SIGHUPing is less likely...
Steinar H. Gunderson [Sun, 14 Apr 2013 16:48:02 +0000 (18:48 +0200)]
Move SIGHUP handling as soon as possible, so that repeated SIGHUPing is less likely to kill us.

7 years agoLog bytes sent and lost better.
Steinar H. Gunderson [Sun, 14 Apr 2013 16:13:19 +0000 (18:13 +0200)]
Log bytes sent and lost better.

7 years agoFix a crash on re-exec if a client was not in SENDING_DATA.
Steinar H. Gunderson [Sun, 14 Apr 2013 16:13:08 +0000 (18:13 +0200)]
Fix a crash on re-exec if a client was not in SENDING_DATA.

7 years agoRename bytes_sent to stream_pos, because that is what it really is.
Steinar H. Gunderson [Sun, 14 Apr 2013 00:30:47 +0000 (02:30 +0200)]
Rename bytes_sent to stream_pos, because that is what it really is.

7 years agoMake more usable documentation.
Steinar H. Gunderson [Sat, 13 Apr 2013 21:03:49 +0000 (23:03 +0200)]
Make more usable documentation.

7 years agoMerge branch 'master' of ssh://git.sesse.net/srv/git.sesse.net/www/cubemap
Steinar H. Gunderson [Sat, 13 Apr 2013 20:53:23 +0000 (22:53 +0200)]
Merge branch 'master' of ssh://git.sesse.net/srv/git.sesse.net/www/cubemap

7 years agoWhen HTTPInput disconnects, also clear the header.
Steinar H. Gunderson [Sat, 13 Apr 2013 20:53:02 +0000 (22:53 +0200)]
When HTTPInput disconnects, also clear the header.

7 years agoAnother include-what-you-use pass.
Steinar H. Gunderson [Sat, 13 Apr 2013 20:48:38 +0000 (22:48 +0200)]
Another include-what-you-use pass.

7 years agoMake backlog_size changeable across HUPs.
Steinar H. Gunderson [Sat, 13 Apr 2013 20:47:24 +0000 (22:47 +0200)]
Make backlog_size changeable across HUPs.

7 years agoMove Server:add_data() into Stream, where it more logically belongs.
Steinar H. Gunderson [Sat, 13 Apr 2013 20:35:45 +0000 (22:35 +0200)]
Move Server:add_data() into Stream, where it more logically belongs.

7 years agoRun include-what-you-use.
Steinar H. Gunderson [Sat, 13 Apr 2013 20:13:56 +0000 (22:13 +0200)]
Run include-what-you-use.

7 years agoFix incorrect struct/class in forward declares (found by Clang).
Steinar H. Gunderson [Sat, 13 Apr 2013 20:12:53 +0000 (22:12 +0200)]
Fix incorrect struct/class in forward declares (found by Clang).

7 years agoMove Client and Stream into their own files.
Steinar H. Gunderson [Sat, 13 Apr 2013 20:02:18 +0000 (22:02 +0200)]
Move Client and Stream into their own files.

7 years agoSupport configurable BACKLOG_SIZE (per-stream). No support for changing across restar...
Steinar H. Gunderson [Sat, 13 Apr 2013 19:53:45 +0000 (21:53 +0200)]
Support configurable BACKLOG_SIZE (per-stream). No support for changing across restarts yet.

7 years agoFix a socket leak in HTTPInput.
Steinar H. Gunderson [Sat, 13 Apr 2013 19:50:13 +0000 (21:50 +0200)]
Fix a socket leak in HTTPInput.

7 years agoFix a spurious warning.
Steinar H. Gunderson [Sat, 13 Apr 2013 17:06:48 +0000 (19:06 +0200)]
Fix a spurious warning.

7 years agoSend data using sendfile() instead of write(). Seems to be something like 30% lower...
Steinar H. Gunderson [Sat, 13 Apr 2013 17:04:27 +0000 (19:04 +0200)]
Send data using sendfile() instead of write(). Seems to be something like 30% lower CPU usage.

7 years agoFix a bug introduced with the move of read_tempfile() to util.cpp.
Steinar H. Gunderson [Sat, 13 Apr 2013 17:03:54 +0000 (19:03 +0200)]
Fix a bug introduced with the move of read_tempfile() to util.cpp.

7 years agoMove the “read the whole file” logic into a new file.
Steinar H. Gunderson [Sat, 13 Apr 2013 16:47:58 +0000 (18:47 +0200)]
Move the “read the whole file” logic into a new file.

7 years agoEven more missing unistd.h includes.
Steinar H. Gunderson [Fri, 12 Apr 2013 23:16:09 +0000 (01:16 +0200)]
Even more missing unistd.h includes.

7 years agoEven more missing unistd.h includes.
Steinar H. Gunderson [Fri, 12 Apr 2013 23:15:57 +0000 (01:15 +0200)]
Even more missing unistd.h includes.

7 years agoThread has virtual member functions, so it should have a virtual destructor.
Steinar H. Gunderson [Fri, 12 Apr 2013 23:15:24 +0000 (01:15 +0200)]
Thread has virtual member functions, so it should have a virtual destructor.

7 years agoFix a missing #include.
Steinar H. Gunderson [Fri, 12 Apr 2013 23:14:32 +0000 (01:14 +0200)]
Fix a missing #include.

7 years agoWhen re-execing, try with --test-config first, and revert if it fails.
Steinar H. Gunderson [Fri, 12 Apr 2013 22:24:36 +0000 (00:24 +0200)]
When re-execing, try with --test-config first, and revert if it fails.

7 years agoImplement --test-config.
Steinar H. Gunderson [Fri, 12 Apr 2013 21:07:01 +0000 (23:07 +0200)]
Implement --test-config.

7 years agoSince we just broke upgrade compatibility, kill some older stuff in the state protos.
Steinar H. Gunderson [Fri, 12 Apr 2013 21:04:50 +0000 (23:04 +0200)]
Since we just broke upgrade compatibility, kill some older stuff in the state protos.

7 years agoSwitch option parsing to getopt.
Steinar H. Gunderson [Fri, 12 Apr 2013 21:04:01 +0000 (23:04 +0200)]
Switch option parsing to getopt.

7 years agoMove some serialization logic into ServerPool, where it belongs.
Steinar H. Gunderson [Fri, 12 Apr 2013 20:31:05 +0000 (22:31 +0200)]
Move some serialization logic into ServerPool, where it belongs.

7 years agoSplit configuration parsing out cleanly from initialization.
Steinar H. Gunderson [Fri, 12 Apr 2013 20:09:24 +0000 (22:09 +0200)]
Split configuration parsing out cleanly from initialization.

This mostly is required for -configtest later, but also generally splits things out
in cleaner and nicer ways than before.

7 years agoFix UDPInput serialization.
Steinar H. Gunderson [Fri, 12 Apr 2013 20:08:09 +0000 (22:08 +0200)]
Fix UDPInput serialization.

7 years agoAdd rudimentary README.
Steinar H. Gunderson [Thu, 11 Apr 2013 23:56:12 +0000 (01:56 +0200)]
Add rudimentary README.

7 years agoSplit config parsing out of parse.h.
Steinar H. Gunderson [Thu, 11 Apr 2013 23:52:29 +0000 (01:52 +0200)]
Split config parsing out of parse.h.

7 years agoSupport UDP input. Also fix some issues with socket closing.
Steinar H. Gunderson [Thu, 11 Apr 2013 21:34:03 +0000 (23:34 +0200)]
Support UDP input. Also fix some issues with socket closing.

7 years agoWhen create_input() fails, give an error message instead of crashing.
Steinar H. Gunderson [Thu, 11 Apr 2013 19:46:10 +0000 (21:46 +0200)]
When create_input() fails, give an error message instead of crashing.

7 years agoDo not keep pending data across HTTP connections.
Steinar H. Gunderson [Thu, 11 Apr 2013 19:36:55 +0000 (21:36 +0200)]
Do not keep pending data across HTTP connections.

7 years agoMore support for multiple input types.
Steinar H. Gunderson [Thu, 11 Apr 2013 19:31:11 +0000 (21:31 +0200)]
More support for multiple input types.

7 years agoMove version identification into a common place.
Steinar H. Gunderson [Thu, 11 Apr 2013 19:08:14 +0000 (21:08 +0200)]
Move version identification into a common place.

7 years agoMake Input a bit more generic, to pave the way for UDP.
Steinar H. Gunderson [Thu, 11 Apr 2013 19:06:03 +0000 (21:06 +0200)]
Make Input a bit more generic, to pave the way for UDP.

7 years agoSupport multiple listening sockets. Actually mostly because it makes the code somewha...
Steinar H. Gunderson [Wed, 10 Apr 2013 23:30:23 +0000 (01:30 +0200)]
Support multiple listening sockets. Actually mostly because it makes the code somewhat more regular.

7 years agoMove stopping into ServerPool.
Steinar H. Gunderson [Wed, 10 Apr 2013 22:24:33 +0000 (00:24 +0200)]
Move stopping into ServerPool.

7 years agoFactor serializing into its own function. Again, less stuff in main().
Steinar H. Gunderson [Wed, 10 Apr 2013 22:21:09 +0000 (00:21 +0200)]
Factor serializing into its own function. Again, less stuff in main().

7 years agoMove stream creation out of main().
Steinar H. Gunderson [Wed, 10 Apr 2013 22:12:42 +0000 (00:12 +0200)]
Move stream creation out of main().

7 years agoSome small refactoring of main().
Steinar H. Gunderson [Wed, 10 Apr 2013 22:06:29 +0000 (00:06 +0200)]
Some small refactoring of main().

7 years agoFactor all the common thread starting/stopping into a common Thread class.
Steinar H. Gunderson [Wed, 10 Apr 2013 21:55:24 +0000 (23:55 +0200)]
Factor all the common thread starting/stopping into a common Thread class.

7 years agoMinor nit.
Steinar H. Gunderson [Wed, 10 Apr 2013 21:45:25 +0000 (23:45 +0200)]
Minor nit.

7 years agoWrap the acceptor into the same thread logic as everything else.
Steinar H. Gunderson [Wed, 10 Apr 2013 21:45:20 +0000 (23:45 +0200)]
Wrap the acceptor into the same thread logic as everything else.

7 years agoMerge branch 'master' of /srv/git.sesse.net/www/cubemap
Steinar H. Gunderson [Wed, 10 Apr 2013 21:36:59 +0000 (23:36 +0200)]
Merge branch 'master' of /srv/git.sesse.net/www/cubemap

7 years agoFactor statistics writing into its own class and file.
Steinar H. Gunderson [Wed, 10 Apr 2013 21:36:56 +0000 (23:36 +0200)]
Factor statistics writing into its own class and file.

7 years agoSmall refactoring in main.cpp.
Steinar H. Gunderson [Wed, 10 Apr 2013 21:35:08 +0000 (23:35 +0200)]
Small refactoring in main.cpp.

7 years agoFactor statistics writing into its own class and file.
Steinar H. Gunderson [Wed, 10 Apr 2013 21:11:00 +0000 (23:11 +0200)]
Factor statistics writing into its own class and file.

7 years agoSeparate out the acceptor stuff into its own file.
Steinar H. Gunderson [Wed, 10 Apr 2013 20:40:00 +0000 (22:40 +0200)]
Separate out the acceptor stuff into its own file.

7 years agoAdd support for setting fwmarks per-connection. Useful for shaping individual streams...
Steinar H. Gunderson [Wed, 10 Apr 2013 19:10:00 +0000 (21:10 +0200)]
Add support for setting fwmarks per-connection. Useful for shaping individual streams (for doing TCP pacing).

7 years agoRemove some obsolete #defines.
Steinar H. Gunderson [Tue, 9 Apr 2013 23:11:22 +0000 (01:11 +0200)]
Remove some obsolete #defines.

7 years agoExplicitly SIGHUP threads to kill them out of syscalls when we want to join them...
Steinar H. Gunderson [Tue, 9 Apr 2013 20:37:55 +0000 (22:37 +0200)]
Explicitly SIGHUP threads to kill them out of syscalls when we want to join them. Also fixes some bugs related to EINTR handling, since they became very obvious now.

7 years agoMeasure how long the serialize/exec/deserialize process takes.
Steinar H. Gunderson [Tue, 9 Apr 2013 20:29:25 +0000 (22:29 +0200)]
Measure how long the serialize/exec/deserialize process takes.

7 years agoSupport writing a stats file listing the number of clients currently connected.
Steinar H. Gunderson [Tue, 9 Apr 2013 20:17:16 +0000 (22:17 +0200)]
Support writing a stats file listing the number of clients currently connected.

7 years agoRename cubemap.cpp to main.cpp. Aaaa....
Steinar H. Gunderson [Tue, 9 Apr 2013 18:49:01 +0000 (20:49 +0200)]
Rename cubemap.cpp to main.cpp. Aaaa....

7 years agoParse the HTTP header (more) properly, and send the headers on to any connecting...
Steinar H. Gunderson [Mon, 8 Apr 2013 23:35:03 +0000 (01:35 +0200)]
Parse the HTTP header (more) properly, and send the headers on to any connecting clients.

7 years agoDeserialize/serialize inputs. Woo, totally glitch-free restarts!
Steinar H. Gunderson [Mon, 8 Apr 2013 23:11:06 +0000 (01:11 +0200)]
Deserialize/serialize inputs. Woo, totally glitch-free restarts!

7 years agoFix a few issues in the to_process() handling.
Steinar H. Gunderson [Mon, 8 Apr 2013 22:48:57 +0000 (00:48 +0200)]
Fix a few issues in the to_process() handling.

7 years agoAdd URL parsing.
Steinar H. Gunderson [Mon, 8 Apr 2013 22:24:31 +0000 (00:24 +0200)]
Add URL parsing.

7 years agoWrite our own HTTP client instead of using curl. Not finished yet (missing URL parsin...
Steinar H. Gunderson [Mon, 8 Apr 2013 22:16:41 +0000 (00:16 +0200)]
Write our own HTTP client instead of using curl. Not finished yet (missing URL parsing, for one).

7 years agoMake most operations on Server deferred, so that we a) do not get bugs with epoll...
Steinar H. Gunderson [Mon, 8 Apr 2013 18:28:03 +0000 (20:28 +0200)]
Make most operations on Server deferred, so that we a) do not get bugs with epoll going out-of-sync with the client state, and b) do not get performance issues stemming from Input not managing to push data often enough to the servers.

7 years agoMake the acceptor thread stop nicely, so that it does not try to mess up the server...
Steinar H. Gunderson [Mon, 8 Apr 2013 18:27:03 +0000 (20:27 +0200)]
Make the acceptor thread stop nicely, so that it does not try to mess up the server while we are shutting down  the server.

7 years agoMake the list of sleeping clients be per-stream instead of global, so we do not wake...
Steinar H. Gunderson [Sun, 7 Apr 2013 22:21:40 +0000 (00:21 +0200)]
Make the list of sleeping clients be per-stream instead of global, so we do not wake up tons of clients for no good reason.

7 years agoIf accept() fails, do not take down the entire thing; it could be a transient error...
Steinar H. Gunderson [Sun, 7 Apr 2013 22:20:51 +0000 (00:20 +0200)]
If accept() fails, do not take down the entire thing; it could be a transient error like “too many open files”. Instead, sleep 0.1 seconds and try again.

7 years agoFix so clients would actually be woken up from the worker thread, not the input threa...
Steinar H. Gunderson [Sun, 7 Apr 2013 21:26:37 +0000 (23:26 +0200)]
Fix so clients would actually be woken up from the worker thread, not the input thread. (Gah!)

7 years agoFix an error where clients in asleep state would hang after a HUP.
Steinar H. Gunderson [Sun, 7 Apr 2013 21:20:16 +0000 (23:20 +0200)]
Fix an error where clients in asleep state would hang after a HUP.

7 years agoMake edge-triggering more consistent.
Steinar H. Gunderson [Sun, 7 Apr 2013 21:13:43 +0000 (23:13 +0200)]
Make edge-triggering more consistent.

7 years agoRemove some leftover code.
Steinar H. Gunderson [Sun, 7 Apr 2013 20:48:13 +0000 (22:48 +0200)]
Remove some leftover code.

7 years agoFix an issue where clients would be put twice into sleeping_clients (probably since...
Steinar H. Gunderson [Sun, 7 Apr 2013 20:06:24 +0000 (22:06 +0200)]
Fix an issue where clients would be put twice into sleeping_clients (probably since the EPOLLOUT trigger would go off once unexpectedly), causing a chain reaction where we did tons of empty write() calls.

7 years agoSome fixes to issues that could cause client hangs.
Steinar H. Gunderson [Sun, 7 Apr 2013 19:10:20 +0000 (21:10 +0200)]
Some fixes to issues that could cause client hangs.

7 years agoStop doing find_stream() every time we send out data; it takes some CPU.
Steinar H. Gunderson [Sun, 7 Apr 2013 19:06:16 +0000 (21:06 +0200)]
Stop doing find_stream() every time we send out data; it takes some CPU.

7 years agoChange from level-triggered to edge-triggered epoll mode. More than halves CPU usage.
Steinar H. Gunderson [Sun, 7 Apr 2013 18:49:54 +0000 (20:49 +0200)]
Change from level-triggered to edge-triggered epoll mode. More than halves CPU usage.

7 years agoHandle streams coming and going from the configuration file across restarts.
Steinar H. Gunderson [Sun, 7 Apr 2013 17:03:42 +0000 (19:03 +0200)]
Handle streams coming and going from the configuration file across restarts.

7 years agoEINTR-protect a bunch of calls, and warn on close() failing.
Steinar H. Gunderson [Sun, 7 Apr 2013 16:51:27 +0000 (18:51 +0200)]
EINTR-protect a bunch of calls, and warn on close() failing.

7 years agoHandle EINTR in epoll_wait().
Steinar H. Gunderson [Sun, 7 Apr 2013 15:46:32 +0000 (17:46 +0200)]
Handle EINTR in epoll_wait().

7 years agoSupport parsing streams from config file. Also support multiple streams (includes...
Steinar H. Gunderson [Sun, 7 Apr 2013 15:42:23 +0000 (17:42 +0200)]
Support parsing streams from config file. Also support multiple streams (includes parsing HTTP), with HTTP error messages and all.

7 years agoWhen a transfer ends, output the URL.
Steinar H. Gunderson [Sun, 7 Apr 2013 15:11:55 +0000 (17:11 +0200)]
When a transfer ends, output the URL.

7 years agoFactor out some config parsing into its own function.
Steinar H. Gunderson [Sun, 7 Apr 2013 15:02:37 +0000 (17:02 +0200)]
Factor out some config parsing into its own function.

7 years agoAdd a sample configuration file.
Steinar H. Gunderson [Sun, 7 Apr 2013 14:11:28 +0000 (16:11 +0200)]
Add a sample configuration file.

7 years agoFetch num_servers from the config file.
Steinar H. Gunderson [Sun, 7 Apr 2013 14:10:39 +0000 (16:10 +0200)]
Fetch num_servers from the config file.