Commit Graph

459 Commits

Author SHA1 Message Date
Brion Vibber
58be61b641 Control channel for queue daemons to request graceful shutdown, restart, or update to listen to a newly added or reconfigured site.
queuectl.php --update -s<site>
  queuectl.php --stop
  queuectl.php --restart

Default control channel is /topic/statusnet-control. For external utilities to send a site update ping direct to the queue server, connect via Stomp and send a message formatted thus:

  update:<nickname>

(Nickname here, *not* server hostname! The rest of the queues will be updated to use nicknames later.)

Note that all currently-connected queue daemons will get these notifications, including both queuedaemon.php and xmppdaemon.php. (XMPP will ignore site update requests for sites that it's not handling.)

Limitations:
* only implemented for stomp queue manager so far
* --update may not yet handle a changed server name properly
* --restart won't reload PHP code files that were already loaded at startup. Still need to stop and restart the daemons from 'outside' when updating code base.
2010-01-26 11:49:49 -08:00
Brion Vibber
ee4ca8f260 quick fix to console.php: don't save blank lines into readline history 2010-01-25 09:41:40 -08:00
Brion Vibber
055a00bcae drop now-unused --skip-xmpp and --xmpp-only options from queuedaemon.php 2010-01-25 09:36:20 -08:00
Brion Vibber
0e852def6a XMPP queued output & initial retooling of DB queue manager to support non-Notice objects.
Queue handlers for XMPP individual & firehose output now send their XML stanzas
to another output queue instead of connecting directly to the chat server. This
lets us have as many general processing threads as we need, while all actual
XMPP input and output go through a single daemon with a single connection open.

This avoids problems with multiple connected resources:
* multiple windows shown in some chat clients (psi, gajim, kopete)
* extra load on server
* incoming message delivery forwarding issues

Database changes:
* queue_item drops 'notice_id' in favor of a 'frame' blob.
  This is based on Craig Andrews' work branch to generalize queues to take any
  object, but conservatively leaving out the serialization for now.
  Table updater (preserves any existing queued items) in db/rc3to09.sql

Code changes to watch out for:
* Queue handlers should now define a handle() method instead of handle_notice()
* QueueDaemon and XmppDaemon now share common i/o (IoMaster) and respawning
  thread management (RespawningDaemon) infrastructure.
* The polling XmppConfirmManager has been dropped, as the message is queued
  directly when saving IM settings.
* Enable $config['queue']['debug_memory'] to output current memory usage at
  each run through the event loop to watch for memory leaks

To do:
* Adapt XMPP i/o to component connection mode for multi-site support.
* XMPP input can also be broken out to a queue, which would allow the actual
  notice save etc to be handled by general queue threads.
* Make sure there are no problems with simply pushing serialized Notice objects
  to queues.
* Find a way to improve interactive performance of the database-backed queue
  handler; polling is pretty painful to XMPP.
* Possibly redo the way QueueHandlers are injected into a QueueManager. The
  grouping used to split out the XMPP output queue is a bit awkward.
2010-01-21 22:40:35 -08:00
Brion Vibber
0bb23e6fd7 drop debug line from xmppdaemon.php, we're done debugging that 2010-01-21 16:34:26 -08:00
Brion Vibber
71df23642c Extra debug instrumentation for xmppdaemon 2010-01-19 12:26:59 -08:00
Brion Vibber
598072468c --xmpp-only hack for queuedaemon.php to run separate queue daemon with only xmpp threads 2010-01-15 11:13:06 -08:00
Brion Vibber
58bc33850a temporary --skip-xmpp flag on queuedaemon.php, allows to run queue daemons but skip subscription to xmpp-based queues
(still working on making these behave gracefully when server is down)
2010-01-14 15:32:37 -08:00
Brion Vibber
35a2f8de4d actually use the user id instead of 0 ;) 2010-01-13 20:46:44 -08:00
Brion Vibber
82f2fe8cca in case we have cached 'fake' inboxes, make sure we rebuild them running initializeinbox.php 2010-01-13 20:30:42 -08:00
Brion Vibber
ec873922e3 Gracefully skip missing user entries when initializing inboxes in bulk 2010-01-13 20:11:00 -08:00
Evan Prodromou
aa23698553 accept file for initializeinbox.php 2010-01-13 18:28:08 -08:00
Siebrand Mazeland
519e1e6250 Update generation of YAML files
* need lower case in some places
* update target path

Not in production yet at translatewiki.net - need some Translate extension code changes to get YAML config support for Gettext first.
2010-01-13 23:33:05 +01:00
Evan Prodromou
b25e59a11f add a script for initializing inboxes 2010-01-13 00:25:43 -08:00
Evan Prodromou
436b8c845e Merge branch '0.9.x' into inblob 2010-01-12 23:53:52 -08:00
Brion Vibber
ec145b73fc Major refactoring of queue handlers to support running multiple sites in one daemon.
Key changes:
* Initialization code moved from common.php to StatusNet class;
  can now switch configurations during runtime.
* As a consequence, configuration files must now be idempotent...
  Be careful with constant, function or class definitions.
* Control structure for daemons/QueueManager/QueueHandler has been refactored;
  the run loop is now managed by IoMaster run via scripts/queuedaemon.php
  IoManager subclasses are woken to handle socket input or polling, and may
  cover multiple sites.
* Plugins can implement notice queue handlers more easily by registering a
  QueueHandler class; no more need to add a daemon.

The new QueueDaemon runs from scripts/queuedaemon.php:

* This replaces most of the old *handler.php scripts; they've been refactored
  to the bare handler classes.
* Spawns multiple child processes to spread load; defaults to CPU count on
  Linux and Mac OS X systems, or override with --threads=N
* When multithreaded, child processes are automatically respawned on failure.
* Threads gracefully shut down and restart when passing a soft memory limit
  (defaults to 90% of memory_limit), limiting damage from memory leaks.
* Support for UDP-based monitoring: http://www.gitorious.org/snqmon

Rough control flow diagram:
QueueDaemon -> IoMaster -> IoManager
                           QueueManager [listen or poll] -> QueueHandler
                           XmppManager [ping & keepalive]
                           XmppConfirmManager [poll updates]

Todo:

* Respawning features not currently available running single-threaded.
* When running single-site, configuration changes aren't picked up.
* New sites or config changes affecting queue subscriptions are not yet
  handled without a daemon restart.
* SNMP monitoring output to integrate with general tools (nagios, ganglia)
* Convert XMPP confirmation message sends to use stomp queue instead of polling
* Convert xmppdaemon.php to IoManager?
* Convert Twitter status, friends import polling daemons to IoManager
* Clean up some error reporting and failure modes
* May need to adjust queue priorities for best perf in backlog/flood cases

Detailed code history available in my daemon-work branch:
http://www.gitorious.org/~brion/statusnet/brion-fixes/commits/daemon-work
2010-01-12 20:45:09 -08:00
Evan Prodromou
ecb3abf84c Merge branch '0.9.x' into inblob 2010-01-11 16:28:27 -08:00
Brion Vibber
be1ac6678d fix long options on deleteuser.php 2010-01-11 13:24:40 -08:00
Siebrand Mazeland
4af6b7f5c3 Lots of tiny message changes.
* Mostly punctuation updates so that the same message is used consistently in all of StatusNet.
* Some cases of "Title Case" removed, because that does not appear to be used consistently.
2010-01-10 12:26:24 +01:00
Evan Prodromou
a180658a3b Merge branch 'inblob' of git@gitorious.org:~evan/statusnet/evans-mainline into inblob
Conflicts:
	classes/Inbox.php
	classes/Notice.php
	classes/Notice_inbox.php
2010-01-09 14:19:59 -08:00
Evan Prodromou
2272438698 remove triminboxes.php; it's no longer used 2010-01-09 10:02:07 -08:00
Evan Prodromou
72934e9f50 Revert "Replace Notice_inbox with Inbox"
We use Notice_inbox to transition to Inbox.

This reverts commit 7640d3f07b.
2010-01-09 10:02:07 -08:00
Evan Prodromou
7ec27b657e Replace Notice_inbox with Inbox 2010-01-09 10:02:07 -08:00
Craig Andrews
055f3fdddb Add an IMAP daemon so StatusNet can process incoming user posts via catch-all mailbox (in addition to the pre-existing script alias method) 2010-01-08 18:52:43 -05:00
Brion Vibber
6f5b765c97 suppress notice for undefined prompt variable when console.php is used from non-interactive terminal 2010-01-06 13:08:56 -08:00
Zach Copley
ff26b8d88b Add an RSSCloud queue handler daemon 2010-01-05 23:19:13 -08:00
Evan Prodromou
0c31c3d80c free some memory in createsim.php 2010-01-04 22:49:09 -10:00
Evan Prodromou
31b62bf781 Merge branch 'batchlocation' 2010-01-04 15:10:43 -10:00
Evan Prodromou
55ba858e8c Script to update the location ID for users
Since we added locations to the database, some users may have
location strings in their profiles but not structured locations. This
script updates the locations for single users or for all users.
2009-12-31 12:38:58 -10:00
Evan Prodromou
7640d3f07b Replace Notice_inbox with Inbox 2009-12-30 09:03:06 -10:00
Brion Vibber
45c9d3d729 Add progress output and optional --sleep-time parameter to triminboxes.php 2009-12-29 14:17:03 -08:00
Evan Prodromou
7160e11395 add setconfig.php script to set configuration options 2009-12-24 15:13:30 -08:00
Evan Prodromou
d40f0931aa remove dead code for handling direct messages, now done with commands 2009-12-15 10:12:36 -05:00
Brion Vibber
b93068083e Clean up console output for non-interactive mode (handy for batch setup scripts) 2009-12-14 11:51:38 -08:00
Brion Vibber
e2f0fc7b3f Make useremail.php executable 2009-12-11 13:14:40 -08:00
Christopher Vollick
280b0b500f Added UserEmail script.
Used to query user's emails.

Mostly used for administration, to see if a user requesting something is who they say.

Also, some people assume that the admin knows this data, and says things like:
"If you could do _____ with the account connected to this email".

It'd be nice if we could do that without raw SQL.
2009-12-11 10:34:57 -05:00
Brion Vibber
4b5e977a7b New _m() gettext wrapper with smart detection of plugin domains. Plugin base class registers your gettext files if present at initialization.
update_pot.sh replaced with update_po_templates.php which can do core, plugins, or all (default).
Top-level Makefile added to build .mo files for plugins as well as core.

As described on list:
http://lists.status.net/pipermail/statusnet-dev/2009-December/002869.html
2009-12-08 12:17:11 -08:00
Brion Vibber
5affa49820 Tweak updateavatarurl.php: emit a newline whether we're on verbose or non-quiet, emit help when no users specified. 2009-12-08 08:30:30 -08:00
Zach Copley
4eceef0dbe Grand ALL permissions to the DB user when setting up a new status.net site. 2009-12-07 11:34:03 -08:00
Brion Vibber
c89b10ffe4 Code style cleanup: dropped some unnecessary =& reference assignments where they're used only out of habit for PHP 4-style object semantics 2009-12-03 12:58:48 -08:00
Siebrand Mazeland
5edc27be6e Do not rebuild/add .mo files by default
FIXME: should be made a command line parameter.
2009-12-03 00:28:00 +01:00
Evan Prodromou
ba4fd15abd add pluginhandler to list of daemons to shut down 2009-12-02 11:07:44 -05:00
Craig Andrews
b5451353e0 make checkschema.php executable (which makes sense, cause the README says to run it :-) ) 2009-11-30 17:00:00 -05:00
Zach Copley
9dc888894b Merge branch 'master' into 0.9.x
* master: (67 commits)
  Ticket 2038: fix bad bug tracker link
  Fix regression in group posting: bug introduced in commit 1319002e15. Need to use actual profile object rather than an id on a variable that doesn't exist when checking blocks :D
  Log database errors when saving notice_inbox entries
  Drop the username from the log id for now; seems to trigger an error loop in some circumstances
  request id on logs... pid + random id per web request + username + method + url
  Add OpenID ini info back into statusnet.ini as a stopgap until we can
  Some changes to the OpenID DataObjects to make them emit the exact same
  OpenID plugin should set 'user_openid.display' as unique key
  Remove relationship: user_openid.user_id -> user.id. I don't think this
  Have OpenID plugin DataObjects emit their own .ini info
  Revert "Allow plugin DB_DataObject classes to not have to use the .ini file by overriding keys(), table(), and sequenceKey() for them"
  Catch and report exceptions from notice_to_omb_notice() instead of letting the OMB queue handler die.
  Fix regression in remote subscription; added hasRole() shadow method on Remote_profile.
  Fix fatal error on OMB subscription for first-timers
  Remove annoying log msg
  Drop error message on setlocale() failure; this is harmless, since we actually have a working locale set up.
  Catch uncaught exception
  Fixed bug where reply-sync bit wasn't getting saved
  Forgot to render the nav menu when on FB Connect login tab
  Facebook plugin no longer takes over Login and Connect settings nav menus
  ...

Conflicts:
	db/08to09_pg.sql
	db/statusnet_pg.sql
	locale/pt_BR/LC_MESSAGES/statusnet.mo
	plugins/Mapstraction/MapstractionPlugin.php
2009-11-30 10:28:58 -08:00
Siebrand Mazeland
99604267d0 Do not use fuzzy in the compiled messages files. 2009-11-21 18:40:34 +01:00
Brion Vibber
09a021cefe start for pluginqueuehandler 2009-11-20 11:20:02 -08:00
Brion Vibber
92453936c2 start for pluginqueuehandler 2009-11-20 11:18:12 -08:00
Zach Copley
4b98edf75f Merge branch '0.9-release'
* 0.9-release: (874 commits)
  Removed call to NewDirectMessage() until IE return is fixed i.e.,
  Don't show flag user button your own profile
  Fixed HXR response for flag user
  Using the right form class name
  Using common_redirect
  Left a form_data class of a <ul> in the user admin panel
  Added validation to fields in user admin panel
  Added a user admin panel
  Added mobile logos for default and identica themes
  Changed gif to png
  Changed this to action. THANKS zach!
  Doing content negotiation only once
  Add execute bit to pingqueuehandler
  Localisation updates for !StatusNet from !translatewiki.net
  Use the browser's geolocation API to set the location on the notice form
  Add geometa library, and include it.
  Add location form elements to the noticeform, and save their values on submission
  Use the $user object nickname, as login name doesnt have to == nickname anymore with plugins such as ldap/etc
  Revert "Re added NICKNAME_FMT constant to router.php."
  Moved most path and server settings to a new paths admin panel
  ...

Conflicts:
	js/util.js
	locale/it_IT/LC_MESSAGES/statusnet.mo
	locale/mk_MK/LC_MESSAGES/statusnet.mo
	locale/mk_MK/LC_MESSAGES/statusnet.po
	locale/pt_BR/LC_MESSAGES/statusnet.mo
	locale/vi_VN/LC_MESSAGES/statusnet.mo
	plugins/InfiniteScroll/infinitescroll.js
	plugins/Realtime/realtimeupdate.js
2009-11-19 20:12:46 -08:00
Zach Copley
ad56ebbb97 Add execute bit to pingqueuehandler 2009-11-19 12:45:45 -08:00
Siebrand Mazeland
69abde6e0c Removing "join-existing". Do not add on update, just rebuild complete pot. The pot file gets really dirty, really quickly. 2009-11-19 01:04:47 +01:00