Commit Graph

228 Commits

Author SHA1 Message Date
Brion Vibber
ec145b73fc Major refactoring of queue handlers to support running multiple sites in one daemon.
Key changes:
* Initialization code moved from common.php to StatusNet class;
  can now switch configurations during runtime.
* As a consequence, configuration files must now be idempotent...
  Be careful with constant, function or class definitions.
* Control structure for daemons/QueueManager/QueueHandler has been refactored;
  the run loop is now managed by IoMaster run via scripts/queuedaemon.php
  IoManager subclasses are woken to handle socket input or polling, and may
  cover multiple sites.
* Plugins can implement notice queue handlers more easily by registering a
  QueueHandler class; no more need to add a daemon.

The new QueueDaemon runs from scripts/queuedaemon.php:

* This replaces most of the old *handler.php scripts; they've been refactored
  to the bare handler classes.
* Spawns multiple child processes to spread load; defaults to CPU count on
  Linux and Mac OS X systems, or override with --threads=N
* When multithreaded, child processes are automatically respawned on failure.
* Threads gracefully shut down and restart when passing a soft memory limit
  (defaults to 90% of memory_limit), limiting damage from memory leaks.
* Support for UDP-based monitoring: http://www.gitorious.org/snqmon

Rough control flow diagram:
QueueDaemon -> IoMaster -> IoManager
                           QueueManager [listen or poll] -> QueueHandler
                           XmppManager [ping & keepalive]
                           XmppConfirmManager [poll updates]

Todo:

* Respawning features not currently available running single-threaded.
* When running single-site, configuration changes aren't picked up.
* New sites or config changes affecting queue subscriptions are not yet
  handled without a daemon restart.
* SNMP monitoring output to integrate with general tools (nagios, ganglia)
* Convert XMPP confirmation message sends to use stomp queue instead of polling
* Convert xmppdaemon.php to IoManager?
* Convert Twitter status, friends import polling daemons to IoManager
* Clean up some error reporting and failure modes
* May need to adjust queue priorities for best perf in backlog/flood cases

Detailed code history available in my daemon-work branch:
http://www.gitorious.org/~brion/statusnet/brion-fixes/commits/daemon-work
2010-01-12 20:45:09 -08:00
Evan Prodromou
2aa0ab9777 let system administrators disallow certain admin panels 2010-01-08 01:00:29 -08:00
Evan Prodromou
505cd382f3 ptiturl.com correct name 2010-01-07 23:38:19 -08:00
Evan Prodromou
2c33e61b94 add default plugins and load them 2010-01-07 16:13:36 -08:00
Craig Andrews
250bcfa8dc Require users to login to view attachments on private sites
Thank you jeff-themovie for this implementation!
2010-01-05 17:49:28 -05:00
Evan Prodromou
cc5534d180 First version of Memcache plugin 2010-01-02 21:16:59 -10:00
Evan Prodromou
39bdda9c7e More configuration options for location sharing 2009-12-28 14:43:34 -08:00
Evan Prodromou
bb93d6b1c7 remove namespace setting from location; it's unused 2009-12-28 14:21:07 -08:00
Evan Prodromou
43ee2ec42f default for login command is disabled 2009-12-14 18:09:30 -05:00
Brion Vibber
3536f01258 Make default & sample val for $config['queue']['queue_basename'] actually work! Now matches the recommendation in README which is in fact correct. Must include the /queue/ prefix (unless somthing else actually makes sense) 2009-12-08 09:57:37 -08:00
Craig Andrews
51f0dd5e37 Add configuration option to toggle the indenting of the output HTML. Defaults to indent enabled. 2009-12-05 01:03:04 -05:00
Brion Vibber
15a9c86979 Ticket 2048: make OMB posting HTTP timeout configurable as $config['omb']['timeout']; defaults to 5 seconds instead of 20-second default in Yadis library 2009-12-01 14:50:18 -08:00
Evan Prodromou
fe18063bd2 Revert "Ask users if they wish to send statistics to SNI, default is off."
This reverts commit 7efea1115f.

Conflicts:

	install.php
2009-11-14 17:38:10 +01:00
Eric Helgeson
7efea1115f Ask users if they wish to send statistics to SNI, default is off.
Users may not know about this setting and are unaware they are sending stats.
This allows them to make that decision.
2009-11-11 18:45:21 -05:00
Brion Vibber
53c86c43c4 Bringing Sphinx search support up to code: broken out to a plugin, now supports multiple sites on a single server.
Upgrade notes:
* Index names have changed from hardcoded 'Identica_people' and 'Identica_notices' to use the database name and actual table names. Must reindex.

New events:
* GetSearchEngine to override default search engine class selection from plugins

New scripts:
* gen_config.php generates a sphinx.conf from database configuration (with theoretical support for status_network table, but it doesn't seem to be cleanly queriable right now without knowing the db setup info for that. Needs generalized support.)
* Replaced old sphinx-indexer.sh and sphinx-cron.sh with index_update.php

Other fixes:
* sphinx.conf.sample better matches our live config, skipping unused stopword list and using a more realistic indexer memory limit

Further notes:
* Probably doesn't work right with PostgreSQL yet; Sphinx can pull from PG but the extraction queries currently look like they use some MySQL-specific functions.
2009-11-10 13:44:40 -08:00
Brion Vibber
5581143bee Rebuilt HTTPClient class as an extension of PEAR HTTP_Request2 package, adding redirect handling and convenience functions.
Caching support will be added in future work after unit tests have been added.

* extlib: add PEAR HTTP_Request2 0.4.1 alpha
* extlib: update PEAR Net_URL2 to 0.3.0 beta for HTTP_Request2 compatibility
* moved direct usage of CURL and file_get_contents to HTTPClient class, excluding external-sourced libraries
* adapted GeonamesPlugin for new HTTPResponse interface

Note some plugins haven't been fully tested yet.
2009-11-02 09:14:15 -08:00
Brion Vibber
b22fc5b74a Revert "Rebuilt HTTPClient class as an extension of PEAR HTTP_Request2 package, adding redirect handling and convenience functions."
Going to restructure a little more before finalizing this...

This reverts commit fa37967858.
2009-11-02 07:51:29 -08:00
Brion Vibber
fa37967858 Rebuilt HTTPClient class as an extension of PEAR HTTP_Request2 package, adding redirect handling and convenience functions.
Caching support will be added in future work after unit tests have been added.

* extlib: add PEAR HTTP_Request2 0.4.1 alpha
* extlib: update PEAR Net_URL2 to 0.3.0 beta for HTTP_Request2 compatibility
* moved direct usage of CURL and file_get_contents to HTTPClient class, excluding external-sourced libraries

Note some plugins haven't been tested yet.
2009-11-02 06:31:28 -08:00
Evan Prodromou
b73d6d18ba Merge branch '0.9.x' into location 2009-10-28 17:24:53 -04:00
Craig Andrews
0b4390e7f2 Make email domain checking optional, as some statusnet installations (such as those behind restrictive corporate firewalls, or on home systems on restrictive connections) cannot connect to any mail
systems, and this check will always fail.
2009-10-26 10:31:12 -04:00
Evan Prodromou
7b227fd1c0 start getting locations from remote services 2009-10-21 22:47:19 -04:00
Evan Prodromou
4e639b8769 Merge branch '0.9.x' of git@gitorious.org:statusnet/mainline into 0.9.x 2009-10-21 22:41:42 -04:00
Evan Prodromou
0df85711d6 reformat default.php 2009-10-21 22:41:23 -04:00
Zach Copley
109a54c4f0 Changed config flag for importing friends' timeline and added some comments 2009-10-17 00:32:02 +00:00
Evan Prodromou
4f5badda94 remove inboxes option 2009-10-13 17:38:27 -04:00
Craig Andrews
f719720e8a Merge remote branch 'statusnet/0.8.x' into 0.9.x
Conflicts:
	lib/common.php
2009-10-11 13:39:10 -04:00
Evan Prodromou
d103522ff3 check the schema 2009-10-01 15:11:12 -04:00
Evan Prodromou
ecb75561af Move configuration defaults to new module and variable
Moved the default values of $config to $default. The code for
setting up $default is moved to lib/default.php.
2009-09-24 22:25:30 -04:00