$config['site']['logperf'] = true; // to record & dump total hits of each type and the runtime to syslog
$config['site']['logperf_detail'] = true; // very verbose -- dump the individual cache keys and queries as they get used (may contain private info in some queries)
Seeing 180 cache gets on a timeline page seems not unusual currently; since these run in serial, even relatively small roundtrip times can add up heavily.
We should consider ways to reduce the number of round trips, such as more frequently storing compound objects or the output of processing in memcached.
Doing parallel multi-key lookups could also help by collapsing round-trip times, but might not be easy to fit into SN's object model. (For things like streams this should actually work pretty well -- grab the list, then when it's returned go grab all the individual items in parallel and return the list)
We can make a lot of HTTP requests from the server side. This change
adds some configuration options for using an HTTP proxy, which can
cache hits from multiple sites (good for status.net-like services, for example).
If a cache entry is dependent on the code that's running, upgrading
(or enabling/disabling plugins) can generate hard-to-track
inconsistencies.
This change adds a close-to-unique fingerprint of the running code to
some cache keys, so that if the fingerprint changes, the old values
are ignored and new values are used.
If the automated uniqueness fails, an administrator can add an extra
config value, $config['site']['build'], that's thrown into the key also.
If a cache entry is dependent on the code that's running, upgrading
(or enabling/disabling plugins) can generate hard-to-track
inconsistencies.
This change adds a close-to-unique fingerprint of the running code to
some cache keys, so that if the fingerprint changes, the old values
are ignored and new values are used.
If the automated uniqueness fails, an administrator can add an extra
config value, $config['site']['build'], that's thrown into the key also.
This option may be useful for intranet sites that don't have direct access to the internet, as they may be unable to successfully fetch those resources.
This will apply to *ALL* plugins in *ALL* languages, so should probably only be used when doing site customization...
You'd probably do:
$config['site']['locale_path'] = '/srv/awesome/data/locale';
$config['plugins']['locale_path'] = '/srv/awesome/data/locale';
with a structure like:
srv/
awesome/
data/
locale/
en/
LC_MESSAGES/
statusnet.po
OpenID.po
AnonymousFave.po
etc, all alongside each other. You could separate plugins from the core if you like.
Where locale files have not already been generated, you can build one for a plugin like so:
php scripts/update_po_templates.php --plugin=MyPlugin
and pull out the template file:
plugins/MyPlugin/locale/MyPlugin.pot
Edit that (make sure you at least set the CHARSET, probably to UTF-8) and save your customized .po
files into the structure as above, and use msgfmt to generate .mo files for final output.
Two prongs here:
* We attempt to enable SNI on the SSL stream context with the appropriate hostname... This requires PHP 5.3.2 and OpenSSL that supports the TLS extensions. Unfortunately this doesn't seem to be working in my testing.
* If set $config['http']['curl'] = true, we'll use the CURL backend if available. In my testing on Ubuntu 10.04, this works. No guarantees on other systems.
I'm not enabling CURL mode by default just yet; want to make sure there's no other surprises.
Currently only one custom theme may be uploaded per site, saved with the name 'custom' and stored into the local/themes subdirectory.
Administrators can upload a .ZIP archive containing a theme through the design admin panel; its contents are validated to ensure that only legit files are saved, and a 5M size quota is enforced.
Theme upload requires the zip extension for PHP; if not present, theme uploading is disabled by default.
Uploading and the custom CSS can be controlled via $config['theme_upload']['enabled'] and $config['custom_css']['enabled'].
Configurable directory/path/server for 'local' subdirectory (currently only as used for themes; local plugins not yet switched over)
Can set $config['local']['dir'] etc; not currently exposed in the admin panels.
Per-site directories on a separate themes server could be set up such as:
$config['local']['dir'] = '/path/to/themes/local/' . $_nickname;
$config['local']['server'] = 'themes.example.com';
$config['local']['path'] = '/local/' . $_nickname;
$config['local']['ssl'] = 'never';
Added a 2-second default timeout for XMLRPC/extended pings, configurable as [ping,timeout].
No longer repeating the entire ping section if we had an HTTP error during a submission.
For now, dropping the bad item and continuing on with others. (Todo: individual retry and cleaner discards of blacklisted broken-for-now sites.)
Users and administrators can set how long an URL can be before it's
shortened, and how long a notice can be before all its URLs are
shortened. They can also turn off shortening altogether.
Squashed commit of the following:
commit d136b39011
Author: Evan Prodromou <evan@status.net>
Date: Mon Apr 26 02:39:00 2010 -0400
use site and user settings to determine when to shorten URLs
commit 1e1c851ff3
Author: Evan Prodromou <evan@status.net>
Date: Mon Apr 26 02:38:40 2010 -0400
add a method to force shortening URLs
commit 4d29ca0b91
Author: Evan Prodromou <evan@status.net>
Date: Mon Apr 26 02:37:41 2010 -0400
static method for getting best URL shortening service
commit a9c6a3bace
Author: Evan Prodromou <evan@status.net>
Date: Mon Apr 26 02:37:11 2010 -0400
allow 0 in numeric entries in othersettings
commit 767ff2f7ec
Author: Evan Prodromou <evan@status.net>
Date: Mon Apr 26 02:36:46 2010 -0400
allow 0 or blank string in inputs
commit 1e21af42a6
Author: Evan Prodromou <evan@status.net>
Date: Mon Apr 26 02:01:11 2010 -0400
add more URL-shortening options to othersettings
commit 869a6be0f5
Author: Evan Prodromou <evan@status.net>
Date: Sat Apr 24 14:22:51 2010 -0400
move url shortener superclass to lib from plugin
commit 9c0c9863d5
Author: Evan Prodromou <evan@status.net>
Date: Sat Apr 24 14:20:28 2010 -0400
documentation and whitespace on UrlShortenerPlugin
commit 7a1dd5798f
Author: Evan Prodromou <evan@status.net>
Date: Sat Apr 24 14:05:46 2010 -0400
add defaults for URL shortening
commit d259c37ad2
Author: Evan Prodromou <evan@status.net>
Date: Sat Apr 24 13:40:10 2010 -0400
Add User_urlshortener_prefs
Add a table for URL shortener prefs, a corresponding class, and the
correct mumbo-jumbo in statusnet.ini to make everything work.
Conflicts:
actions/imsettings.php
lib/jabber.php
Made a quick attempt to merge the new JID validation into the XmppPlugin, have not had a chance to test that version live yet.
Should also move over the test cases.
I swapped the settings from negative to positive ($config['queue']['stomp_transactions'] = false rather than $config['queue']['stomp_no_transactions'] = true), gave them defaults (both on for best ActiveMQ experience), and added notes to the README about configuring them.
- switch 'en_US' to 'en', fixes the "admin panel switches to Arabic" bug
- tweak setting descriptions to clarify that most of the time we'll be using browser language
- add a backend switch to disable language detection (should this be exposed to ui?)
We've been making pretty crummy tag: URIs for a while. We should
continue to favor HTTP URIs, since it's nice to be able to discover
things about an object you've shared the ID of. Where that's not
possible, this makes nicer tag URIs.
All breakout queues that we're going to need to listen to now need to be explicitly listed in $config['queue']['breakout'].
Until XMPP is moved to component model, this setting will let the individual processes work with their own queues:
$config['queue']['breakout'][] = 'xmpp/xmppout/' . $config['site']['nickname'];
- Multiplexing queues into groups and for multiple sites.
- Sharing vs breakout configurable per site and per queue via $config['queue']['breakout']
- Detect how many times a message is redelivered, discard if it's killed too many daemons
- count configurable with $config['queue']['max_retries']
- can dump the items to files in $config['queue']['dead_letter_dir']
Queue daemon memory & resource leak fixes:
- avoid unnecessary reconnections to memcached server (switch persistent connections back in on second initialization, assuming it's child process)
- monkey-patch for leaky .ini loads in DB_DataObject::databaseStructure() - was leaking 200k per active switch
- applied leak fixes to Status_network as well, using intermediate base Safe_DataObject for both it and Memcache_DataObject
Misc queue fixes:
- correct handling of child processes exiting due to signal termination instead of regular exit
- shutdown instead of infinite respawn loop if we're already past the soft memory limit at startup
- Added --all option for xmppdaemon... still opens one xmpp connection per site that has xmpp active
Cache updates:
- add Cache::increment() method with native support for memcached atomic increment
Defaulting to only looking at last 90 days of activity, can be adjusted up or down.
$config['tag']['cutoff'] = 86400 * 90;
$config['popular']['cutoff'] = 86400 * 90;
Per-user and per-group tag clouds do not use the cutoff (and it doesn't help with indexing on them).
Adds a robots.txt file to the site root. Defaults defined by
'robotstxt' section of config. New events StartRobotsTxt and
EndRobotsTxt to let plugins add information. Probably not
useful if path is not /, but won't hurt anything, either.
If an array of multiple servers is put in $config['queue']['stomp_server'], enqueues will pick a random server to send to (failing over automatically if any are down).
Queue handling daemons connect all servers so they get events no matter where they were delivered.
In case of disconnection, daemons should now handle it gracefully and attempt to reconnect every 60 seconds or so, automatically resubscribing to all queues once it's back up.
Can put to 'native' failover for reads as well by disabling $config['stomp']['manual_failover'] = false; but this is untested and may explode in addition to requiring that your ActiveMQ cluster actually be set up to handle its own data distribution.
Additionally, can choose which queues to mark as persistent by setting $config['stomp']['persistent'] to an array of queue names.
* testing: (130 commits)
HTTP auth provided is evaluated even if it's not required
Rename rc3to09.sql to rc3torc4.sql to avoid confusion if we add a last-minute change after this!
Add new oauth tables and modifications to 'consumer' table for rc4
Centred leaderboard ad
camelcase the uap param names
move leaderboard to after the header
Moved rectangle ad into aside and leaderboard to the right in header.
Aligning wide skyscraper to the right instead of left
CSS ids and classes fixed in UAPPlugin
wrong height for rectangle in BlankAd
Add the moved BlankAdPlugin
make BlankAd dir and change to use a 1x1 image
move BlankAdPlugin to its own dir
Add BlankAdPlugin to test ad layout in different themes
make uapplugin an abstract class
move UAP plugin to core
Lowercased switch cases in UAP Plugin
Plugin for Universal Ad Package. Outputs four most widely used ad types.
Add persistent:true property to Stomp messages so ActiveMQ doesn't decide to discard them even though persistence is enabled on the broker. :) (Thanks Aric!)
quick fix: use common_path() on realtime update JS so it works with the new JS path code (will pull from main server for now)
...
Conflicts:
actions/apioauthaccesstoken.php
actions/apioauthauthorize.php
actions/apioauthrequesttoken.php
actions/editapplication.php
actions/newapplication.php
lib/apiauth.php
lib/queuemanager.php
lib/router.php
We have about 10-12 JavaScript pages per Web page. They usually
are based on the same server as the Web pages, but since they're
static files, it makes sense to offload them to a lite server that
handles static files well.
This commit lets you set a separate Javascript server and path for the
default Javascript code in StatusNet.
Squashed commit of the following:
commit 139d1622fdafe5ad00c820224416d9021efc3234
Author: Evan Prodromou <evan@status.net>
Date: Wed Jan 27 11:30:24 2010 -0500
modules that call htmloutputter::script() don't prescribe js/ path
commit c6ca3174af73efed55eaed5ff1e2a3bdc77d2d87
Author: Evan Prodromou <evan@status.net>
Date: Wed Jan 27 11:28:07 2010 -0500
configurable server and path for javascript files
queuectl.php --update -s<site>
queuectl.php --stop
queuectl.php --restart
Default control channel is /topic/statusnet-control. For external utilities to send a site update ping direct to the queue server, connect via Stomp and send a message formatted thus:
update:<nickname>
(Nickname here, *not* server hostname! The rest of the queues will be updated to use nicknames later.)
Note that all currently-connected queue daemons will get these notifications, including both queuedaemon.php and xmppdaemon.php. (XMPP will ignore site update requests for sites that it's not handling.)
Limitations:
* only implemented for stomp queue manager so far
* --update may not yet handle a changed server name properly
* --restart won't reload PHP code files that were already loaded at startup. Still need to stop and restart the daemons from 'outside' when updating code base.
