Commit Graph

92 Commits

Author SHA1 Message Date
Evan Prodromou
9f0715a993 Merge branch '0.9.x' into 1.0.x 2010-08-03 16:05:03 -07:00
Siebrand Mazeland
e753422480 * Address i18n related FIXMEs after talk with Brion.
* Tweak message
2010-07-30 19:15:07 +02:00
Siebrand Mazeland
e7acb45b57 * add string for translation with translator documentation
* add FIXMEs for strings that may need i18n, but leaving decision to other dev(s)
2010-07-29 12:58:48 +02:00
Evan Prodromou
d73feb82d8 cache sitemap notice and user counts for 4h 2010-07-14 10:38:34 -04:00
Brion Vibber
004200f958 Merge branch '0.9.x' of gitorious.org:statusnet/mainline into 1.0.x 2010-06-11 12:09:55 -07:00
Evan Prodromou
6d39a75137 use DB_DataObject_Cast objects in a couple of different places in the code 2010-06-04 15:29:38 -04:00
Evan Prodromou
f1ea678aae memcache_dataobject supports some DB_DataObject_Cast objects as values 2010-06-04 12:52:05 -04:00
Evan Prodromou
727ea5a516 Merge branch '0.9.x' into 1.0.x 2010-04-07 10:47:29 -04:00
Evan Prodromou
76cce8a3c5 Merge branch 'master' into 0.9.x 2010-04-07 10:46:23 -04:00
Evan Prodromou
08745e4eac Merge branch 'master' into testing 2010-04-07 10:45:44 -04:00
Evan Prodromou
82326a3b69 show PEAR_Error message, not the object 2010-04-07 10:43:14 -04:00
Brion Vibber
01a03e34c8 Merge branch '0.9.x' into 1.0.x 2010-03-29 15:15:51 -07:00
Brion Vibber
d44e5ac935 Add $config['db']['annotate_queries'] option to include caller ID comments into queries. Comment can then be seen in process list, slow query logs on the server, aiding in tracking down unexpected slow queries.
SELECT /* queuedaemon.php Ostatus_profile->processPost */ * FROM notice WHERE ( notice.uri = 'http://stormcloud.local/mublog2/notice/479' )
INSERT /* POST Notice::saveNew */ INTO notice (profile_id , content ....
2010-03-29 13:14:11 -07:00
Brion Vibber
abf2ce873b Avoid notice when reporting DB errors for objects that don't have an 'id' field 2010-03-24 14:18:25 -07:00
Brion Vibber
2aaf8d4e30 Add class and (if present) id to DB_DataObject error exceptions; often they're VERRRRRY vague, and it helps to know what type of item is failing! 2010-02-24 01:09:12 +00:00
Brion Vibber
81b6b58e33 Merge branch 'master' into testing
Conflicts:
	lib/stompqueuemanager.php
2010-02-16 09:22:02 -08:00
Brion Vibber
c74aea589d Stomp queue restructuring for mass scalability:
- Multiplexing queues into groups and for multiple sites.
- Sharing vs breakout configurable per site and per queue via $config['queue']['breakout']
- Detect how many times a message is redelivered, discard if it's killed too many daemons
 - count configurable with $config['queue']['max_retries']
 - can dump the items to files in $config['queue']['dead_letter_dir']

Queue daemon memory & resource leak fixes:
- avoid unnecessary reconnections to memcached server (switch persistent connections back in on second initialization, assuming it's child process)
- monkey-patch for leaky .ini loads in DB_DataObject::databaseStructure() - was leaking 200k per active switch
- applied leak fixes to Status_network as well, using intermediate base Safe_DataObject for both it and Memcache_DataObject

Misc queue fixes:
- correct handling of child processes exiting due to signal termination instead of regular exit
- shutdown instead of infinite respawn loop if we're already past the soft memory limit at startup
- Added --all option for xmppdaemon... still opens one xmpp connection per site that has xmpp active

Cache updates:
- add Cache::increment() method with native support for memcached atomic increment
2010-02-16 09:16:51 -08:00
Brion Vibber
875e1a70ce Don't spew warnings on usage of MEMCACHE_COMPRESSED constant when memcache PHP extension is not present.
Switched to a locally-defined Cache::COMPRESSED, translating that to MEMCACHE_COMPRESSED in the plugin.
2010-02-05 09:47:56 -08:00
Evan Prodromou
f6eecf02fc add simple cache getter/setter static functions to Memcached_DataObject 2010-01-29 15:01:21 -05:00
Evan Prodromou
513f8be07a hide most DB_DataObject errors 2010-01-28 16:26:55 -05:00
Evan Prodromou
d5f86f7a8a Merge branch 'master' of git@gitorious.org:statusnet/mainline 2010-01-28 13:54:19 -05:00
Evan Prodromou
84b5480007 update mysqltimestamps on insert and update 2010-01-28 13:54:01 -05:00
Brion Vibber
ffaaf9de4a Don't preemptively close existing DB connections for web views (needed to keep # of conns from going insane on multi-site queue daemons, so just doing for CLI)
May, or may not, help with mystery session problems
2010-01-27 23:51:22 -08:00
Evan Prodromou
663e4e02a1 Merge branch 'master' into testing
Conflicts:
	lib/queuemanager.php
2010-01-25 18:13:09 -05:00
Evan Prodromou
e26a843caf Offload inbox updates to a queue handler to speed up posting online
Moved much of the writing that happens when posting a notice to a new
queuehandler, distribqueuehandler. This updates tags, groups, replies
and inboxes at queue time (or at Web time, if queues are disabled).

To make this work well, I had to break up the monolithic
Notice::blowCaches() and make cache blowing happen closer to where
data is updated.

Squashed commit of the following:

commit 5257626c62750ac4ac1db0ce2b71410c5711cfa3
Author: Evan Prodromou <evan@status.net>
Date:   Mon Jan 25 14:56:41 2010 -0500

    slightly better handling of blowing tag memory cache

commit 8a22a3cdf6ec28685da129a0313e7b2a0837c9ef
Author: Evan Prodromou <evan@status.net>
Date:   Mon Jan 25 01:42:56 2010 -0500

    change 'distribute' to 'distrib' so not too long for dbqueue

commit 7a063315b0f7fad27cb6fbd2bdd74e253af83e4f
Author: Evan Prodromou <evan@status.net>
Date:   Mon Jan 25 01:39:15 2010 -0500

    change handle_notice() to handle() in distributqueuehandler

commit 1a39ccd28b9994137d7bfd21bb4f230546938e77
Author: Evan Prodromou <evan@status.net>
Date:   Mon Jan 25 16:05:25 2010 -0500

    error with queuemanager

commit e6b3bb93f305cfd2de71a6340b8aa6fb890049b7
Author: Evan Prodromou <evan@status.net>
Date:   Mon Jan 25 01:11:34 2010 -0500

    Blow memcache at different point rather than one big function for Notice class

commit 94d557cdc016187d1d0647ae1794cd94d6fb8ac8
Author: Evan Prodromou <evan@status.net>
Date:   Mon Jan 25 00:48:44 2010 -0500

    Blow memcache at different point rather than one big function for Notice class

commit 1c781dd08c88a35dafc5c01230b4872fd6b95182
Author: Evan Prodromou <evan@status.net>
Date:   Wed Jan 20 08:54:18 2010 -0500

    move broadcasting and distributing to new queuehandler

commit da3e46d26b84e4f028f34a13fd2ee373e4c1b954
Author: Evan Prodromou <evan@status.net>
Date:   Wed Jan 20 08:53:12 2010 -0500

    Move distribution of notices to new distribute queue handler
2010-01-25 18:08:21 -05:00
Evan Prodromou
e6cf293db8 Recover caching logic lost in bad merge
I made a bad merge on Jan 10th from master to 0.9.x. This lost a number
of memcache enhancements made on the 0.9.x branch. I've been able to
re-do the manual merge, and this represents the changes. Most of them
are related to caching on insert.
2010-01-24 22:39:32 -05:00
Evan Prodromou
29d83c8ca9 Merge branch 'testing' of git@gitorious.org:statusnet/mainline into testing 2010-01-22 14:18:21 -05:00
Evan Prodromou
a3e484a0e8 Memcached_DataObject unnecessarily require_once'd itself 2010-01-22 10:46:11 -05:00
Brion Vibber
8dafe09ab2 Option to log slow db queries or all db queries
$config['db']['log_queries'] = true; // all
$config['db']['log_slow_queries'] = 10; // queries taking > 10 seconds
2010-01-21 11:07:52 -08:00
Brion Vibber
ec145b73fc Major refactoring of queue handlers to support running multiple sites in one daemon.
Key changes:
* Initialization code moved from common.php to StatusNet class;
  can now switch configurations during runtime.
* As a consequence, configuration files must now be idempotent...
  Be careful with constant, function or class definitions.
* Control structure for daemons/QueueManager/QueueHandler has been refactored;
  the run loop is now managed by IoMaster run via scripts/queuedaemon.php
  IoManager subclasses are woken to handle socket input or polling, and may
  cover multiple sites.
* Plugins can implement notice queue handlers more easily by registering a
  QueueHandler class; no more need to add a daemon.

The new QueueDaemon runs from scripts/queuedaemon.php:

* This replaces most of the old *handler.php scripts; they've been refactored
  to the bare handler classes.
* Spawns multiple child processes to spread load; defaults to CPU count on
  Linux and Mac OS X systems, or override with --threads=N
* When multithreaded, child processes are automatically respawned on failure.
* Threads gracefully shut down and restart when passing a soft memory limit
  (defaults to 90% of memory_limit), limiting damage from memory leaks.
* Support for UDP-based monitoring: http://www.gitorious.org/snqmon

Rough control flow diagram:
QueueDaemon -> IoMaster -> IoManager
                           QueueManager [listen or poll] -> QueueHandler
                           XmppManager [ping & keepalive]
                           XmppConfirmManager [poll updates]

Todo:

* Respawning features not currently available running single-threaded.
* When running single-site, configuration changes aren't picked up.
* New sites or config changes affecting queue subscriptions are not yet
  handled without a daemon restart.
* SNMP monitoring output to integrate with general tools (nagios, ganglia)
* Convert XMPP confirmation message sends to use stomp queue instead of polling
* Convert xmppdaemon.php to IoManager?
* Convert Twitter status, friends import polling daemons to IoManager
* Clean up some error reporting and failure modes
* May need to adjust queue priorities for best perf in backlog/flood cases

Detailed code history available in my daemon-work branch:
http://www.gitorious.org/~brion/statusnet/brion-fixes/commits/daemon-work
2010-01-12 20:45:09 -08:00
Brion Vibber
4cc9b183d7 Avoid spewing warnings about returning by reference on memcached cache misses by setting a var and returning it instead of trying to return false directly.
This hacky workaround is brought to you by DB_DataObject's PHP 4 roots.
2010-01-12 12:20:45 -08:00
Brion Vibber
f397d35527 Regression fix: don't spew notices to log every time we get a non-cached user object 2010-01-11 13:24:52 -08:00
Evan Prodromou
9d3893255a don't put Users with object IDs in the cache, and don't fetch them 2010-01-10 12:31:43 -08:00
Brion Vibber
78214c4e06 Exclude process-specific link & result cache references from serialized Memcached_Data_Object instances.
Should fix seemingly-random bugs due to destructor free()ing local resources by mistake.

cherry-pick from 0.9.x
2010-01-04 14:38:56 -08:00
Brion Vibber
8f02379f6e Revert "Take Memcached_DataObject destructor back out to check whether it might be causing some under-the-hood problems."
This reverts commit 89cca01259.
2010-01-04 14:37:39 -08:00
Evan Prodromou
d32fb7c7c4 return false from Memcached_DataObject::staticGet() on not found, like DB_DataObject 2010-01-02 16:28:46 -10:00
Evan Prodromou
2bd32dfad7 change harmless check of to check of in Memcached_DataObject 2010-01-02 16:21:19 -10:00
Evan Prodromou
d2bbf65f19 Merge branch 'master' of git@gitorious.org:statusnet/mainline 2010-01-01 11:27:28 -10:00
Evan Prodromou
79c2e3f720 Revert "add cleanup method to cleanup a single row"
This reverts commit b0527801d9.
2010-01-01 11:27:13 -10:00
Evan Prodromou
b0527801d9 add cleanup method to cleanup a single row 2010-01-01 11:02:56 -10:00
Evan Prodromou
727978332a stop using DB_DataObject's staticGet() which caches 2010-01-01 10:57:22 -10:00
Brion Vibber
89cca01259 Take Memcached_DataObject destructor back out to check whether it might be causing some under-the-hood problems. 2009-12-30 16:42:57 -08:00
Brion Vibber
ff50c2b91d Revert debugging code 2009-12-30 16:40:59 -08:00
Brion Vibber
9218cce3cd subbug debug info to check on free ordering 2009-12-30 15:55:15 -08:00
Evan Prodromou
f6bf952980 Merge branch 'testing' 2009-12-22 16:44:19 -08:00
Evan Prodromou
22f02b35ad call DB_DataObject::__destruct() if it exists 2009-12-15 12:38:15 -05:00
Evan Prodromou
945661d942 take out DB_DataObject destructor 2009-12-15 12:33:17 -05:00
Brion Vibber
b9040a7cc4 Add destructor on Memcached_DataObject to free DB_DataObject's global storage for an object when that object itself is destroyed.
Reduces some, but not all, memory leakage for long-running processes.
2009-12-14 16:36:01 -08:00
Brion Vibber
727357695e Debug check to track down live error -- wrong data type sometimes being sent down to Memcached_DataObject::cacheKey() via various fetch functions, need a backtrace to track it down. 2009-12-11 14:19:18 -08:00
Brion Vibber
53c86c43c4 Bringing Sphinx search support up to code: broken out to a plugin, now supports multiple sites on a single server.
Upgrade notes:
* Index names have changed from hardcoded 'Identica_people' and 'Identica_notices' to use the database name and actual table names. Must reindex.

New events:
* GetSearchEngine to override default search engine class selection from plugins

New scripts:
* gen_config.php generates a sphinx.conf from database configuration (with theoretical support for status_network table, but it doesn't seem to be cleanly queriable right now without knowing the db setup info for that. Needs generalized support.)
* Replaced old sphinx-indexer.sh and sphinx-cron.sh with index_update.php

Other fixes:
* sphinx.conf.sample better matches our live config, skipping unused stopword list and using a more realistic indexer memory limit

Further notes:
* Probably doesn't work right with PostgreSQL yet; Sphinx can pull from PG but the extraction queries currently look like they use some MySQL-specific functions.
2009-11-10 13:44:40 -08:00