Commit Graph

71 Commits

Author SHA1 Message Date
Evan Prodromou
ed99fd51e8 don't save statuses from protected users 2010-09-07 04:00:57 -04:00
Zach Copley
5b7b38ab66 Some fixes to make the twitterstatusfetcher behave better in a multi-site configuration 2010-03-27 23:43:05 +00:00
root
982edc653f Another typo 2010-03-04 23:30:36 -05:00
Zach Copley
72f72d17db - Fix cache handling in TwitterStatusFetcher
- Other stability fixes
2010-02-04 01:53:08 +00:00
Evan Prodromou
436b8c845e Merge branch '0.9.x' into inblob 2010-01-12 23:53:52 -08:00
Brion Vibber
ec145b73fc Major refactoring of queue handlers to support running multiple sites in one daemon.
Key changes:
* Initialization code moved from common.php to StatusNet class;
  can now switch configurations during runtime.
* As a consequence, configuration files must now be idempotent...
  Be careful with constant, function or class definitions.
* Control structure for daemons/QueueManager/QueueHandler has been refactored;
  the run loop is now managed by IoMaster run via scripts/queuedaemon.php
  IoManager subclasses are woken to handle socket input or polling, and may
  cover multiple sites.
* Plugins can implement notice queue handlers more easily by registering a
  QueueHandler class; no more need to add a daemon.

The new QueueDaemon runs from scripts/queuedaemon.php:

* This replaces most of the old *handler.php scripts; they've been refactored
  to the bare handler classes.
* Spawns multiple child processes to spread load; defaults to CPU count on
  Linux and Mac OS X systems, or override with --threads=N
* When multithreaded, child processes are automatically respawned on failure.
* Threads gracefully shut down and restart when passing a soft memory limit
  (defaults to 90% of memory_limit), limiting damage from memory leaks.
* Support for UDP-based monitoring: http://www.gitorious.org/snqmon

Rough control flow diagram:
QueueDaemon -> IoMaster -> IoManager
                           QueueManager [listen or poll] -> QueueHandler
                           XmppManager [ping & keepalive]
                           XmppConfirmManager [poll updates]

Todo:

* Respawning features not currently available running single-threaded.
* When running single-site, configuration changes aren't picked up.
* New sites or config changes affecting queue subscriptions are not yet
  handled without a daemon restart.
* SNMP monitoring output to integrate with general tools (nagios, ganglia)
* Convert XMPP confirmation message sends to use stomp queue instead of polling
* Convert xmppdaemon.php to IoManager?
* Convert Twitter status, friends import polling daemons to IoManager
* Clean up some error reporting and failure modes
* May need to adjust queue priorities for best perf in backlog/flood cases

Detailed code history available in my daemon-work branch:
http://www.gitorious.org/~brion/statusnet/brion-fixes/commits/daemon-work
2010-01-12 20:45:09 -08:00
Evan Prodromou
2db840b2f5 use Inbox in twitterstatusfetcher 2010-01-09 14:02:01 -08:00
Zach Copley
eb0a5977ff Don't try to broadcast the notice if something went wrong. 2009-12-04 20:39:50 +00:00
Zach Copley
cff12603cf Make imported Twitter notices show up via real time plugins. 2009-12-04 20:17:42 +00:00
Zach Copley
fcf1c4e53f Remove useless debugging statement 2009-12-02 19:33:08 +00:00
Brion Vibber
088081675f Revert "Remove more contractions"
This reverts commit 5ab709b739.

Missed this one yesterday...
2009-11-09 20:01:46 +01:00
Siebrand Mazeland
b10f362ede Merge branch '0.9.x' of git://gitorious.org/statusnet/mainline into 0.9.x 2009-11-08 23:33:58 +01:00
Siebrand Mazeland
5ab709b739 Remove more contractions
* doesn't
* won't
* isn't
* don't
2009-11-08 23:32:15 +01:00
Brion Vibber
fc5002015b Revert "* [Cc]an't -> [Cc]annot"
This reverts commit 0ab17f382b.
2009-11-08 23:28:51 +01:00
Siebrand Mazeland
0ab17f382b * [Cc]an't -> [Cc]annot
* [Cc]ould't -> [Cc]ould not
2009-11-08 23:22:38 +01:00
Brion Vibber
5581143bee Rebuilt HTTPClient class as an extension of PEAR HTTP_Request2 package, adding redirect handling and convenience functions.
Caching support will be added in future work after unit tests have been added.

* extlib: add PEAR HTTP_Request2 0.4.1 alpha
* extlib: update PEAR Net_URL2 to 0.3.0 beta for HTTP_Request2 compatibility
* moved direct usage of CURL and file_get_contents to HTTPClient class, excluding external-sourced libraries
* adapted GeonamesPlugin for new HTTPResponse interface

Note some plugins haven't been fully tested yet.
2009-11-02 09:14:15 -08:00
Brion Vibber
b22fc5b74a Revert "Rebuilt HTTPClient class as an extension of PEAR HTTP_Request2 package, adding redirect handling and convenience functions."
Going to restructure a little more before finalizing this...

This reverts commit fa37967858.
2009-11-02 07:51:29 -08:00
Brion Vibber
fa37967858 Rebuilt HTTPClient class as an extension of PEAR HTTP_Request2 package, adding redirect handling and convenience functions.
Caching support will be added in future work after unit tests have been added.

* extlib: add PEAR HTTP_Request2 0.4.1 alpha
* extlib: update PEAR Net_URL2 to 0.3.0 beta for HTTP_Request2 compatibility
* moved direct usage of CURL and file_get_contents to HTTPClient class, excluding external-sourced libraries

Note some plugins haven't been tested yet.
2009-11-02 06:31:28 -08:00
Zach Copley
0fd8e758ad Make queuing and daemons work via events 2009-10-14 04:50:16 +00:00
Zach Copley
9d87313eae Merge branch '0.9.x' into pluginize-twitter-bridge
Conflicts:
	plugins/TwitterBridge/twitterauthorization.php
2009-09-08 16:02:57 -07:00
Zach Copley
5efe588174 Moved the rest of the Twitter stuff into the TwitterBridge plugin 2009-08-26 00:59:06 +00:00