Major refactoring of queue handlers to support running multiple sites in one daemon.

Key changes:
* Initialization code moved from common.php to StatusNet class;
  can now switch configurations during runtime.
* As a consequence, configuration files must now be idempotent...
  Be careful with constant, function or class definitions.
* Control structure for daemons/QueueManager/QueueHandler has been refactored;
  the run loop is now managed by IoMaster run via scripts/queuedaemon.php
  IoManager subclasses are woken to handle socket input or polling, and may
  cover multiple sites.
* Plugins can implement notice queue handlers more easily by registering a
  QueueHandler class; no more need to add a daemon.

The new QueueDaemon runs from scripts/queuedaemon.php:

* This replaces most of the old *handler.php scripts; they've been refactored
  to the bare handler classes.
* Spawns multiple child processes to spread load; defaults to CPU count on
  Linux and Mac OS X systems, or override with --threads=N
* When multithreaded, child processes are automatically respawned on failure.
* Threads gracefully shut down and restart when passing a soft memory limit
  (defaults to 90% of memory_limit), limiting damage from memory leaks.
* Support for UDP-based monitoring: http://www.gitorious.org/snqmon

Rough control flow diagram:
QueueDaemon -> IoMaster -> IoManager
                           QueueManager [listen or poll] -> QueueHandler
                           XmppManager [ping & keepalive]
                           XmppConfirmManager [poll updates]

Todo:

* Respawning features not currently available running single-threaded.
* When running single-site, configuration changes aren't picked up.
* New sites or config changes affecting queue subscriptions are not yet
  handled without a daemon restart.
* SNMP monitoring output to integrate with general tools (nagios, ganglia)
* Convert XMPP confirmation message sends to use stomp queue instead of polling
* Convert xmppdaemon.php to IoManager?
* Convert Twitter status, friends import polling daemons to IoManager
* Clean up some error reporting and failure modes
* May need to adjust queue priorities for best perf in backlog/flood cases

Detailed code history available in my daemon-work branch:
http://www.gitorious.org/~brion/statusnet/brion-fixes/commits/daemon-work
This commit is contained in:
Brion Vibber
2010-01-12 19:57:15 -08:00
parent 2b10e359fe
commit ec145b73fc
39 changed files with 2612 additions and 1203 deletions

52
plugins/Facebook/facebookqueuehandler.php Executable file → Normal file
View File

@@ -1,4 +1,3 @@
#!/usr/bin/env php
<?php
/*
* StatusNet - the distributed open-source microblogging tool
@@ -18,21 +17,9 @@
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
define('INSTALLDIR', realpath(dirname(__FILE__) . '/../..'));
if (!defined('STATUSNET') && !defined('LACONICA')) { exit(1); }
$shortoptions = 'i::';
$longoptions = array('id::');
$helptext = <<<END_OF_FACEBOOK_HELP
Daemon script for pushing new notices to Facebook.
-i --id Identity (default none)
END_OF_FACEBOOK_HELP;
require_once INSTALLDIR . '/scripts/commandline.inc';
require_once INSTALLDIR . '/plugins/Facebook/facebookutil.php';
require_once INSTALLDIR . '/lib/queuehandler.php';
class FacebookQueueHandler extends QueueHandler
{
@@ -41,33 +28,24 @@ class FacebookQueueHandler extends QueueHandler
return 'facebook';
}
function start()
function handle_notice($notice)
{
$this->log(LOG_INFO, "INITIALIZE");
if ($this->_isLocal($notice)) {
return facebookBroadcastNotice($notice);
}
return true;
}
function handle_notice($notice)
/**
* Determine whether the notice was locally created
*
* @param Notice $notice the notice
*
* @return boolean locality
*/
function _isLocal($notice)
{
return facebookBroadcastNotice($notice);
return ($notice->is_local == Notice::LOCAL_PUBLIC ||
$notice->is_local == Notice::LOCAL_NONPUBLIC);
}
function finish()
{
}
}
if (have_option('i')) {
$id = get_option_value('i');
} else if (have_option('--id')) {
$id = get_option_value('--id');
} else if (count($args) > 0) {
$id = $args[0];
} else {
$id = null;
}
$handler = new FacebookQueueHandler($id);
$handler->runOnce();