diff --git a/INSTALL b/INSTALL index 54225ca315..75237fa1fe 100644 --- a/INSTALL +++ b/INSTALL @@ -106,9 +106,9 @@ especially if you've previously installed PHP/MySQL packages. 1. Unpack the tarball you downloaded on your Web server. Usually a command like this will work: - tar zxf statusnet-1.0.1.tar.gz + tar zxf statusnet-1.1.0-alpha1.tar.gz - ...which will make a statusnet-1.0.1 subdirectory in your current + ...which will make a statusnet-1.1.0-alpha1 subdirectory in your current directory. (If you don't have shell access on your Web server, you may have to unpack the tarball on your local computer and FTP the files to the server.) @@ -116,7 +116,7 @@ especially if you've previously installed PHP/MySQL packages. 2. Move the tarball to a directory of your choosing in your Web root directory. Usually something like this will work: - mv statusnet-1.0.1 /var/www/statusnet + mv statusnet-1.1.0-alpha1 /var/www/statusnet This will make your StatusNet instance available in the statusnet path of your server, like "http://example.net/statusnet". "microblog" or diff --git a/README b/README index d08f9ff1ab..b58aaeebc8 100644 --- a/README +++ b/README @@ -2,7 +2,7 @@ README ------ -StatusNet 1.0.1 +StatusNet 1.1.0-alpha1 3 October 2011 This is the README file for StatusNet, the Open Source social @@ -107,46 +107,23 @@ for additional terms. New this version ================ -This is a minor bug fix release since 1.0.0, released 30 September -2011. It fixes the following bugs: +This is a minor bug fix and feature release since 1.0.1, released 3 +October 2011. (Because the plugin interface has changed in an upwardly +compatible way, we've incremented the minor version number. See +http://semver.org/ for the semantic versioning scheme we follow.) -- Change default OEmbed provider from oohembed to noembed. -- Fix problem with path matching on new installs. +It includes the following changes: -Notable additions in the 1.0.x series: +- ActivitySpam plugin to check updates against spamicity.info +- Options to hide users who've been silenced or have posted spammy updates. +- OfflineBackup experimental plugin. +- Fixes for TwitterBridge to correctly handle replies through the bridge. +- Improvements in ActivityStreams JSON output to better match 1.0 spec. +- Console scripts for managing groups. +- Bug fix for conversation counts in conversation streams. +- Rights for moderators to manage spam. -- Support for private updates, including private-to-groups, private - within a site, and private to followers only. -- Conversation mode in streams; notices appear along with all replies. -- Microapps -- post different types of activities to timelines, with - interaction. Events, bookmarks, Q&A, and polls included by default. -- New 3-column layout in 'neo' theme by default. Older, 2-column layout - themes have been removed. -- Alphabetical, searchable user directory. -- Alphabetical, searchable group directory. -- Groups can require all posts to be private ('private groups'), and - limit members to the group. -- Users can make all posts private to their followers ('private stream'), - and require authorization to follow. -- General plugin for IM support; added AIM, IRC and MSN to existing - XMPP code. -- Support for Twitter-like lists, to follow other users without - interfering with the home timeline. -- Subscription to searches. -- Subscription to tags. -- Drupal-style schema system ("schemax") allows in-place database - upgrades from various software versions. -- Fine-grained control of URL shortening, and an internal URL shortener - available. -- Extended profile for private, enterprise sites. -- sites are private by default. -- Blog plugin for extended posts. -- Plugin to restrict all users of a site to a single email domain. -- Plugin to send a daily email summary to site users. -- Deeper integration with Activity Streams (http://activitystrea.ms) format. -- Automated upgrade script. - -A full changelog is available at http://status.net/wiki/StatusNet_1.0.1. +A full changelog is available at http://status.net/wiki/StatusNet_1.1.0-alpha1. Troubleshooting =============== diff --git a/UPGRADE b/UPGRADE index 17838185f3..c50e7622ff 100644 --- a/UPGRADE +++ b/UPGRADE @@ -1,7 +1,7 @@ Upgrading ========= -If you've been using StatusNet 0.9.9 or lower, or if you've +If you've been using StatusNet 1.0 or lower, or if you've been tracking the "git" version of the software, you will probably want to upgrade and keep your existing data. Try these step-by-step instructions; read to the end first before trying them. @@ -24,7 +24,7 @@ instructions; read to the end first before trying them. 5. Once all writing processes to your site are turned off, make a final backup of the Web directory and database. 6. Move your StatusNet directory to a backup spot, like "statusnet.bak". -7. Unpack your StatusNet 1.0.1 tarball and move it to "statusnet" or +7. Unpack your StatusNet 1.1.0-alpha1 tarball and move it to "statusnet" or wherever your code used to be. 8. Copy the config.php file and the contents of the avatar/, background/, file/, and local/ subdirectories from your old directory to your new @@ -37,7 +37,7 @@ instructions; read to the end first before trying them. reversed. YOU CAN EASILY DESTROY YOUR SITE WITH THIS STEP. Don't do it without a known-good backup! - In your new StatusNet 1.0.1 directory and AFTER YOU MAKE A + In your new StatusNet 1.1.0-alpha1 directory and AFTER YOU MAKE A BACKUP run the upgrade.php script like this: php ./scripts/upgrade.php diff --git a/lib/framework.php b/lib/framework.php index e7a1829bc6..dc195c65d7 100644 --- a/lib/framework.php +++ b/lib/framework.php @@ -19,13 +19,13 @@ if (!defined('STATUSNET') && !defined('LACONICA')) { exit(1); } -define('STATUSNET_BASE_VERSION', '1.0.1'); -define('STATUSNET_LIFECYCLE', ''); // 'dev', 'alpha[0-9]+', 'beta[0-9]+', 'rc[0-9]+', 'release' -define('STATUSNET_VERSION', STATUSNET_BASE_VERSION . STATUSNET_LIFECYCLE); +define('STATUSNET_BASE_VERSION', '1.1.0'); +define('STATUSNET_LIFECYCLE', 'alpha1'); // 'dev', 'alpha[0-9]+', 'beta[0-9]+', 'rc[0-9]+', 'release' +define('STATUSNET_VERSION', STATUSNET_BASE_VERSION . '-' . STATUSNET_LIFECYCLE); define('LACONICA_VERSION', STATUSNET_VERSION); // compatibility -define('STATUSNET_CODENAME', 'Get It Together'); +define('STATUSNET_CODENAME', 'Fight for Your Right'); define('AVATAR_PROFILE_SIZE', 96); define('AVATAR_STREAM_SIZE', 48); @@ -34,7 +34,6 @@ define('AVATAR_MINI_SIZE', 24); define('NOTICES_PER_PAGE', 20); define('PROFILES_PER_PAGE', 20); define('MESSAGES_PER_PAGE', 20); -define('GROUPS_PER_PAGE', 20); define('FOREIGN_NOTICE_SEND', 1); define('FOREIGN_NOTICE_RECV', 2); @@ -170,10 +169,9 @@ function PEAR_ErrorToPEAR_Exception($err) } if ($err->getCode()) { - throw new PEAR_Exception($msg, $err, $err->getCode()); - } else { - throw new PEAR_Exception($msg, $err); + throw new PEAR_Exception($err->getMessage(), $err->getCode()); } + throw new PEAR_Exception($err->getMessage()); } PEAR::setErrorHandling(PEAR_ERROR_CALLBACK, 'PEAR_ErrorToPEAR_Exception'); diff --git a/plugins/ActivitySpam/ActivitySpamPlugin.php b/plugins/ActivitySpam/ActivitySpamPlugin.php new file mode 100644 index 0000000000..a905e72cca --- /dev/null +++ b/plugins/ActivitySpam/ActivitySpamPlugin.php @@ -0,0 +1,318 @@ +. + * + * @category Spam + * @package StatusNet + * @author Evan Prodromou + * @copyright 2011,2012 StatusNet, Inc. + * @license http://www.fsf.org/licensing/licenses/agpl-3.0.html AGPL 3.0 + * @link http://status.net/ + */ + +if (!defined('STATUSNET')) { + // This check helps protect against security problems; + // your code file can't be executed directly from the web. + exit(1); +} + +/** + * Check new notices with activity spam service. + * + * @category Spam + * @package StatusNet + * @author Evan Prodromou + * @copyright 2011,2012 StatusNet, Inc. + * @license http://www.fsf.org/licensing/licenses/agpl-3.0.html AGPL 3.0 + * @link http://status.net/ + */ +class ActivitySpamPlugin extends Plugin +{ + public $server = null; + public $hideSpam = false; + + const REVIEWSPAM = 'ActivitySpamPlugin::REVIEWSPAM'; + const TRAINSPAM = 'ActivitySpamPlugin::TRAINSPAM'; + + /** + * Initializer + * + * @return boolean hook value; true means continue processing, false means stop. + */ + function initialize() + { + $this->filter = new SpamFilter(common_config('activityspam', 'server'), + common_config('activityspam', 'consumerkey'), + common_config('activityspam', 'secret')); + + $this->hideSpam = common_config('activityspam', 'hidespam'); + + return true; + } + + /** + * Database schema setup + * + * @see Schema + * @see ColumnDef + * + * @return boolean hook value; true means continue processing, false means stop. + */ + + function onCheckSchema() + { + $schema = Schema::get(); + $schema->ensureTable('spam_score', Spam_score::schemaDef()); + + Spam_score::upgrade(); + + return true; + } + + /** + * Load related modules when needed + * + * @param string $cls Name of the class to be loaded + * + * @return boolean hook value; true means continue processing, false means stop. + */ + + function onAutoload($cls) + { + $dir = dirname(__FILE__); + + switch ($cls) + { + case 'TrainAction': + case 'SpamAction': + include_once $dir . '/' . strtolower(mb_substr($cls, 0, -6)) . '.php'; + return false; + case 'Spam_score': + include_once $dir . '/'.$cls.'.php'; + return false; + case 'SpamFilter': + case 'SpamNoticeStream': + case 'TrainSpamForm': + case 'TrainHamForm': + include_once $dir . '/'.strtolower($cls).'.php'; + return false; + default: + return true; + } + } + + /** + * When a notice is saved, check its spam score + * + * @param Notice $notice Notice that was just saved + * + * @return boolean hook value; true means continue processing, false means stop. + */ + + function onEndNoticeSave($notice) + { + try { + + $result = $this->filter->test($notice); + + $score = Spam_score::saveNew($notice, $result); + + $this->log(LOG_INFO, "Notice " . $notice->id . " has spam score " . $score->score); + + } catch (Exception $e) { + // Log but continue + $this->log(LOG_ERR, $e->getMessage()); + } + + return true; + } + + function onNoticeDeleteRelated($notice) { + $score = Spam_score::staticGet('notice_id', $notice->id); + if (!empty($score)) { + $score->delete(); + } + return true; + } + + function onUserRightsCheck($profile, $right, &$result) { + switch ($right) { + case self::REVIEWSPAM: + case self::TRAINSPAM: + $result = ($profile->hasRole(Profile_role::MODERATOR) || $profile->hasRole('modhelper')); + return false; + default: + return true; + } + } + + function onGetSpamFilter(&$filter) { + $filter = $this->filter; + return false; + } + + function onEndShowNoticeOptionItems($nli) + { + $profile = Profile::current(); + + if (!empty($profile) && $profile->hasRight(self::TRAINSPAM)) { + + $notice = $nli->getNotice(); + $out = $nli->getOut(); + + if (!empty($notice)) { + + $score = $this->getScore($notice); + + if (empty($score)) { + $this->debug("No score for notice " . $notice->id); + // XXX: show a question-mark or something + } else if ($score->is_spam) { + $form = new TrainHamForm($out, $notice); + $form->show(); + } else if (!$score->is_spam) { + $form = new TrainSpamForm($out, $notice); + $form->show(); + } + } + } + + return true; + } + + /** + * Map URLs to actions + * + * @param Net_URL_Mapper $m path-to-action mapper + * + * @return boolean hook value; true means continue processing, false means stop. + */ + + function onRouterInitialized($m) + { + $m->connect('main/train/spam', + array('action' => 'train', 'category' => 'spam')); + $m->connect('main/train/ham', + array('action' => 'train', 'category' => 'ham')); + $m->connect('main/spam', + array('action' => 'spam')); + return true; + } + + function onEndShowStyles($action) + { + $action->element('style', null, + '.form-train-spam input.submit { background: url('.$this->path('icons/bullet_black.png').') no-repeat 0px 0px } ' . "\n" . + '.form-train-ham input.submit { background: url('.$this->path('icons/exclamation.png').') no-repeat 0px 0px } '); + return true; + } + + function onEndPublicGroupNav($nav) + { + $user = common_current_user(); + + if (!empty($user) && $user->hasRight(self::REVIEWSPAM)) { + $nav->out->menuItem(common_local_url('spam'), + _m('MENU','Spam'), + // TRANS: Menu item title in search group navigation panel. + _('Notices marked as spam'), + $nav->actionName == 'spam', + 'nav_timeline_spam'); + } + + return true; + } + + function onPluginVersion(&$versions) + { + $versions[] = array('name' => 'ActivitySpam', + 'version' => STATUSNET_VERSION, + 'author' => 'Evan Prodromou', + 'homepage' => 'http://status.net/wiki/Plugin:ActivitySpam', + 'description' => + _m('Test notices against the Activity Spam service.')); + return true; + } + + function getScore($notice) + { + $score = Spam_score::staticGet('notice_id', $notice->id); + + if (!empty($score)) { + return $score; + } + + try { + + $result = $this->filter->test($notice); + + $score = Spam_score::saveNew($notice, $result); + + $this->log(LOG_INFO, "Notice " . $notice->id . " has spam score " . $score->score); + + } catch (Exception $e) { + // Log but continue + $this->log(LOG_ERR, $e->getMessage()); + $score = null; + } + + return $score; + } + + function onStartReadWriteTables(&$alwaysRW, &$rwdb) { + $alwaysRW[] = 'spam_score'; + return true; + } + + + function onEndNoticeInScope($notice, $profile, &$bResult) + { + if ($this->hideSpam) { + if ($bResult) { + + $score = Spam_score::staticGet('notice_id', $notice->id); + + if (!empty($score) && $score->is_spam) { + if (empty($profile) || + ($profile->id !== $notice->profile_id && + !$profile->hasRight(self::REVIEWSPAM))) { + $bResult = false; + } + } + } + } + + return true; + } + + /** + * Pre-cache our spam scores if needed. + */ + function onEndNoticeListPrefill(&$notices, &$profiles, $avatarSize) { + if ($this->hideSpam) { + foreach ($notices as $notice) { + $ids[] = $notice->id; + } + Memcached_DataObject::multiGet('Spam_score', 'notice_id', $ids); + } + return true; + } +} diff --git a/plugins/ActivitySpam/Spam_score.php b/plugins/ActivitySpam/Spam_score.php new file mode 100644 index 0000000000..997a9f83ad --- /dev/null +++ b/plugins/ActivitySpam/Spam_score.php @@ -0,0 +1,216 @@ +. + * + * @category Spam + * @package StatusNet + * @author Evan Prodromou + * @copyright 2011 StatusNet, Inc. + * @license http://www.fsf.org/licensing/licenses/agpl-3.0.html AGPL 3.0 + * @link http://status.net/ + */ + +if (!defined('STATUSNET')) { + exit(1); +} + +/** + * Score of a notice per the activity spam service + * + * @category Spam + * @package StatusNet + * @author Evan Prodromou + * @license http://www.fsf.org/licensing/licenses/agpl.html AGPLv3 + * @link http://status.net/ + * + * @see DB_DataObject + */ + +class Spam_score extends Managed_DataObject +{ + const MAX_SCALE = 10000; + public $__table = 'spam_score'; // table name + + public $notice_id; // int + public $score; // float + public $created; // datetime + + /** + * Get an instance by key + * + * @param string $k Key to use to lookup (usually 'notice_id' for this class) + * @param mixed $v Value to lookup + * + * @return Spam_score object found, or null for no hits + * + */ + function staticGet($k, $v=null) + { + return Managed_DataObject::staticGet('Spam_score', $k, $v); + } + + function saveNew($notice, $result) { + + $score = new Spam_score(); + + $score->notice_id = $notice->id; + $score->score = $result->probability; + $score->is_spam = $result->isSpam; + $score->scaled = Spam_score::scale($score->score); + $score->created = common_sql_now(); + $score->notice_created = $notice->created; + + $score->insert(); + + self::blow('spam_score:notice_ids'); + + return $score; + } + + function save($notice, $result) { + + $orig = null; + $score = Spam_score::staticGet('notice_id', $notice->id); + + if (empty($score)) { + $score = new Spam_score(); + } else { + $orig = clone($score); + } + + $score->notice_id = $notice->id; + $score->score = $result->probability; + $score->is_spam = $result->isSpam; + $score->scaled = Spam_score::scale($score->score); + $score->created = common_sql_now(); + $score->notice_created = $notice->created; + + if (empty($orig)) { + $score->insert(); + } else { + $score->update($orig); + } + + self::blow('spam_score:notice_ids'); + + return $score; + } + + function delete() + { + self::blow('spam_score:notice_ids'); + self::blow('spam_score:notice_ids;last'); + parent::delete(); + } + + /** + * The One True Thingy that must be defined and declared. + */ + public static function schemaDef() + { + return array( + 'description' => 'score of the notice per activityspam', + 'fields' => array( + 'notice_id' => array('type' => 'int', + 'not null' => true, + 'description' => 'notice getting scored'), + 'score' => array('type' => 'double', + 'not null' => true, + 'description' => 'score for the notice (0.0, 1.0)'), + 'scaled' => array('type' => 'int', + 'description' => 'scaled score for the notice (0, 10000)'), + 'is_spam' => array('type' => 'tinyint', + 'description' => 'flag for spamosity'), + 'created' => array('type' => 'datetime', + 'not null' => true, + 'description' => 'date this record was created'), + 'notice_created' => array('type' => 'datetime', + 'description' => 'date the notice was created'), + ), + 'primary key' => array('notice_id'), + 'foreign keys' => array( + 'spam_score_notice_id_fkey' => array('notice', array('notice_id' => 'id')), + ), + 'indexes' => array( + 'spam_score_created_idx' => array('created'), + 'spam_score_scaled_idx' => array('scaled'), + ), + ); + } + + public static function upgrade() + { + Spam_score::upgradeScaled(); + Spam_score::upgradeIsSpam(); + Spam_score::upgradeNoticeCreated(); + } + + protected static function upgradeScaled() + { + $score = new Spam_score(); + $score->whereAdd('scaled IS NULL'); + + if ($score->find()) { + while ($score->fetch()) { + $orig = clone($score); + $score->scaled = Spam_score::scale($score->score); + $score->update($orig); + } + } + } + + protected static function upgradeIsSpam() + { + $score = new Spam_score(); + $score->whereAdd('is_spam IS NULL'); + + if ($score->find()) { + while ($score->fetch()) { + $orig = clone($score); + $score->is_spam = ($score->score >= 0.90) ? 1 : 0; + $score->update($orig); + } + } + } + + protected static function upgradeNoticeCreated() + { + $score = new Spam_score(); + $score->whereAdd('notice_created IS NULL'); + + if ($score->find()) { + while ($score->fetch()) { + $notice = Notice::staticGet('id', $score->notice_id); + if (!empty($notice)) { + $orig = clone($score); + $score->notice_created = $notice->created; + $score->update($orig); + } + } + } + } + + public static function scale($score) + { + $raw = round($score * Spam_score::MAX_SCALE); + return max(0, min(Spam_score::MAX_SCALE, $raw)); + } +} diff --git a/plugins/ActivitySpam/icons/bullet_black.png b/plugins/ActivitySpam/icons/bullet_black.png new file mode 100644 index 0000000000..57619706d1 Binary files /dev/null and b/plugins/ActivitySpam/icons/bullet_black.png differ diff --git a/plugins/ActivitySpam/icons/exclamation.png b/plugins/ActivitySpam/icons/exclamation.png new file mode 100644 index 0000000000..c37bd062e6 Binary files /dev/null and b/plugins/ActivitySpam/icons/exclamation.png differ diff --git a/plugins/ActivitySpam/scripts/testuser.php b/plugins/ActivitySpam/scripts/testuser.php new file mode 100644 index 0000000000..357e04a7c2 --- /dev/null +++ b/plugins/ActivitySpam/scripts/testuser.php @@ -0,0 +1,105 @@ +. + */ + +define('INSTALLDIR', realpath(dirname(__FILE__) . '/../../../..')); + +$shortoptions = 'i:n:a'; +$longoptions = array('id=', 'nickname=', 'all'); + +$helptext = <<orderBy('created'); + $user->limit($offset, $limit); + + $found = $user->find(); + + if ($found) { + while ($user->fetch()) { + try { + testUser($filter, $user); + } catch (Exception $e) { + printfnq("ERROR testing user %s\n: %s", $user->nickname, $e->getMessage()); + } + } + $offset += $found; + } + + } while ($found > 0); +} + +function testUser($filter, $user) { + + printfnq("Testing user %s\n", $user->nickname); + + $profile = Profile::staticGet('id', $user->id); + + $str = new ProfileNoticeStream($profile, $profile); + + $offset = 0; + $limit = 100; + + do { + $notice = $str->getNotices($offset, $limit); + while ($notice->fetch()) { + try { + printfv("Testing notice %d...", $notice->id); + $result = $filter->test($notice); + Spam_score::save($notice, $result); + printfv("%s\n", ($result->isSpam) ? "SPAM" : "HAM"); + } catch (Exception $e) { + printfnq("ERROR testing notice %d: %s\n", $notice->id, $e->getMessage()); + } + } + $offset += $notice->N; + } while ($notice->N > 0); +} + +try { + $filter = null; + Event::handle('GetSpamFilter', array(&$filter)); + if (empty($filter)) { + throw new Exception(_("No spam filter.")); + } + if (have_option('a', 'all')) { + testAllUsers($filter); + } else { + $user = getUser(); + testUser($filter, $user); + } +} catch (Exception $e) { + print $e->getMessage()."\n"; + exit(1); +} diff --git a/plugins/ActivitySpam/scripts/trainuser.php b/plugins/ActivitySpam/scripts/trainuser.php new file mode 100644 index 0000000000..3399e751ba --- /dev/null +++ b/plugins/ActivitySpam/scripts/trainuser.php @@ -0,0 +1,81 @@ +. + */ + +define('INSTALLDIR', realpath(dirname(__FILE__) . '/../../../..')); + +$shortoptions = 'i:n:t:'; +$longoptions = array('id=', 'nickname=', 'category='); + +$helptext = <<nickname); + + $profile = Profile::staticGet('id', $user->id); + + $str = new ProfileNoticeStream($profile, $profile); + + $offset = 0; + $limit = 100; + + do { + $notice = $str->getNotices($offset, $limit); + while ($notice->fetch()) { + try { + printfv("Training notice %d...", $notice->id); + $filter->trainOnError($notice, $category); + $result = $filter->test($notice); + $score = Spam_score::save($notice, $result); + printfv("%s\n", ($result->isSpam) ? "SPAM" : "HAM"); + } catch (Exception $e) { + printfnq("ERROR training notice %d\n: %s", $notice->id, $e->getMessage()); + } + } + $offset += $notice->N; + } while ($notice->N > 0); +} + +try { + $filter = null; + Event::handle('GetSpamFilter', array(&$filter)); + if (empty($filter)) { + throw new Exception(_("No spam filter.")); + } + $user = getUser(); + $category = get_option_value('t', 'category'); + if ($category !== SpamFilter::HAM && + $category !== SpamFilter::SPAM) { + throw new Exception(_("No such category.")); + } + trainUser($filter, $user, $category); +} catch (Exception $e) { + print $e->getMessage()."\n"; + exit(1); +} diff --git a/plugins/ActivitySpam/spam.php b/plugins/ActivitySpam/spam.php new file mode 100644 index 0000000000..a66b73a829 --- /dev/null +++ b/plugins/ActivitySpam/spam.php @@ -0,0 +1,165 @@ +. + * + * @category Spam + * @package StatusNet + * @author Evan Prodromou + * @copyright 2012 StatusNet, Inc. + * @license http://www.fsf.org/licensing/licenses/agpl-3.0.html AGPL 3.0 + * @link http://status.net/ + */ + +if (!defined('STATUSNET')) { + // This check helps protect against security problems; + // your code file can't be executed directly from the web. + exit(1); +} + +require_once INSTALLDIR.'/lib/noticelist.php'; + +/** + * SpamAction + * + * Shows the latest spam on the service + * + * @category Spam + * @package StatusNet + * @author Evan Prodromou + * @copyright 2012 StatusNet, Inc. + * @license http://www.fsf.org/licensing/licenses/agpl-3.0.html AGPL 3.0 + * @link http://status.net/ + */ + +class SpamAction extends Action +{ + var $page = null; + var $notices = null; + + function title() { + return _("Latest Spam"); + } + + /** + * For initializing members of the class. + * + * @param array $argarray misc. arguments + * + * @return boolean true + */ + + function prepare($argarray) + { + parent::prepare($argarray); + + $this->page = ($this->arg('page')) ? ($this->arg('page')+0) : 1; + + // User must be logged in. + + $user = common_current_user(); + + if (empty($user)) { + throw new ClientException(_("You must be logged in to review."), 403); + } + + // User must have the right to review spam + + if (!$user->hasRight(ActivitySpamPlugin::REVIEWSPAM)) { + throw new ClientException(_('You cannot review spam on this site.'), 403); + } + + $stream = new SpamNoticeStream($user->getProfile()); + + $this->notices = $stream->getNotices(($this->page-1)*NOTICES_PER_PAGE, + NOTICES_PER_PAGE + 1); + + if($this->page > 1 && $this->notices->N == 0) { + throw new ClientException(_('No such page.'), 404); + } + + return true; + } + + /** + * Handler method + * + * @param array $argarray is ignored since it's now passed in in prepare() + * + * @return void + */ + + function handle($argarray=null) + { + parent::handle($args); + + $this->showPage(); + } + + /** + * Fill the content area + * + * Shows a list of the notices in the public stream, with some pagination + * controls. + * + * @return void + */ + + function showContent() + { + $nl = new NoticeList($this->notices, $this); + + $cnt = $nl->show(); + + if ($cnt == 0) { + $this->showEmptyList(); + } + + $this->pagination($this->page > 1, + $cnt > NOTICES_PER_PAGE, + $this->page, + 'spam'); + } + + function showEmptyList() + { + // TRANS: Text displayed for public feed when there are no public notices. + $message = _('This is the timeline of spam messages for %%site.name%% but none have been detected yet.'); + + $this->elementStart('div', 'guide'); + $this->raw(common_markup_to_html($message)); + $this->elementEnd('div'); + } + + /** + * Return true if read only. + * + * MAY override + * + * @param array $args other arguments + * + * @return boolean is read only action? + */ + + function isReadOnly($args) + { + return true; + } +} diff --git a/plugins/ActivitySpam/spamfilter.php b/plugins/ActivitySpam/spamfilter.php new file mode 100644 index 0000000000..3ddfdad039 --- /dev/null +++ b/plugins/ActivitySpam/spamfilter.php @@ -0,0 +1,171 @@ +. + * + * @category Spam + * @package StatusNet + * @author Evan Prodromou + * @copyright 2012 StatusNet, Inc. + * @license http://www.fsf.org/licensing/licenses/agpl-3.0.html AGPL 3.0 + * @link http://status.net/ + */ + +if (!defined('STATUSNET')) { + // This check helps protect against security problems; + // your code file can't be executed directly from the web. + exit(1); +} + +/** + * Spam filter class + * + * Local proxy for remote filter + * + * @category Spam + * @package StatusNet + * @author Evan Prodromou + * @copyright 2012 StatusNet, Inc. + * @license http://www.fsf.org/licensing/licenses/agpl-3.0.html AGPL 3.0 + * @link http://status.net/ + */ + +class SpamFilter extends OAuthClient { + + const HAM = 'ham'; + const SPAM = 'spam'; + + public $server; + + function __construct($server, $consumerKey, $secret) { + parent::__construct($consumerKey, $secret); + $this->server = $server; + } + + protected function toActivity($notice) { + // FIXME: need this to autoload ActivityStreamsMediaLink + $doc = new ActivityStreamJSONDocument(); + + $activity = $notice->asActivity(null); + + return $activity; + } + + public function test($notice) { + + $activity = $this->toActivity($notice); + return $this->testActivity($activity); + } + + public function testActivity($activity) { + + $response = $this->postJSON($this->server . "/is-this-spam", $activity->asArray()); + + $result = json_decode($response->getBody()); + + return $result; + } + + public function train($notice, $category) { + + $activity = $this->toActivity($notice); + return $this->trainActivity($activity, $category); + + } + + public function trainActivity($activity, $category) { + + switch ($category) { + case self::HAM: + $endpoint = '/this-is-ham'; + break; + case self::SPAM: + $endpoint = '/this-is-spam'; + break; + default: + throw new Exception("Unknown category: " + $category); + } + + $response = $this->postJSON($this->server . $endpoint, $activity->asArray()); + + // We don't do much with the results + return true; + } + + public function trainOnError($notice, $category) { + + $activity = $this->toActivity($notice); + + return $this->trainActivityOnError($activity, $category); + } + + public function trainActivityOnError($activity, $category) { + + $result = $this->testActivity($activity); + + if (($category === self::SPAM && $result->isSpam) || + ($category === self::HAM && !$result->isSpam)) { + return true; + } else { + return $this->trainActivity($activity, $category); + } + } + + function postJSON($url, $body) + { + $request = OAuthRequest::from_consumer_and_token($this->consumer, + $this->token, + 'POST', + $url); + + $request->sign_request($this->sha1_method, + $this->consumer, + $this->token); + + $hclient = new HTTPClient($url); + + $hclient->setConfig(array('connect_timeout' => 120, + 'timeout' => 120, + 'follow_redirects' => true, + 'ssl_verify_peer' => false, + 'ssl_verify_host' => false)); + + $hclient->setMethod(HTTP_Request2::METHOD_POST); + $hclient->setBody(json_encode($body)); + $hclient->setHeader('Content-Type', 'application/json'); + $hclient->setHeader($request->to_header()); + + // Twitter is strict about accepting invalid "Expect" headers + // No reason not to clear it still here -ESP + + $hclient->setHeader('Expect', ''); + + try { + $response = $hclient->send(); + $code = $response->getStatus(); + if (!$response->isOK()) { + throw new OAuthClientException($response->getBody(), $code); + } + return $response; + } catch (Exception $e) { + throw new OAuthClientException($e->getMessage(), $e->getCode()); + } + } +} diff --git a/plugins/ActivitySpam/spamnoticestream.php b/plugins/ActivitySpam/spamnoticestream.php new file mode 100644 index 0000000000..ffb8d08025 --- /dev/null +++ b/plugins/ActivitySpam/spamnoticestream.php @@ -0,0 +1,101 @@ +. + * + * @category Spam + * @package StatusNet + * @author Evan Prodromou + * @copyright 2012 StatusNet, Inc. + * @license http://www.fsf.org/licensing/licenses/agpl-3.0.html AGPL 3.0 + * @link http://status.net/ + */ + +if (!defined('STATUSNET')) { + // This check helps protect against security problems; + // your code file can't be executed directly from the web. + exit(1); +} + +/** + * Spam notice stream + * + * @category Spam + * @package StatusNet + * @author Evan Prodromou + * @copyright 2012 StatusNet, Inc. + * @license http://www.fsf.org/licensing/licenses/agpl-3.0.html AGPL 3.0 + * @link http://status.net/ + */ + +class SpamNoticeStream extends ScopingNoticeStream +{ + function __construct($tag, $profile = -1) + { + if (is_int($profile) && $profile == -1) { + $profile = Profile::current(); + } + parent::__construct(new CachingNoticeStream(new RawSpamNoticeStream(), + 'spam_score:notice_ids')); + } +} + +/** + * Raw stream of spammy notices + * + * @category Stream + * @package StatusNet + * @author Evan Prodromou + * @copyright 2011 StatusNet, Inc. + * @license http://www.fsf.org/licensing/licenses/agpl-3.0.html AGPL 3.0 + * @link http://status.net/ + */ + +class RawSpamNoticeStream extends NoticeStream +{ + function getNoticeIds($offset, $limit, $since_id, $max_id) + { + $ss = new Spam_score(); + + $ss->is_spam = 1; + + $ss->selectAdd(); + $ss->selectAdd('notice_id'); + + Notice::addWhereSinceId($ss, $since_id, 'notice_id'); + Notice::addWhereMaxId($ss, $max_id, 'notice_id'); + + $ss->orderBy('notice_created DESC, notice_id DESC'); + + if (!is_null($offset)) { + $ss->limit($offset, $limit); + } + + $ids = array(); + + if ($ss->find()) { + while ($ss->fetch()) { + $ids[] = $ss->notice_id; + } + } + + return $ids; + } +} diff --git a/plugins/ActivitySpam/train.php b/plugins/ActivitySpam/train.php new file mode 100644 index 0000000000..f5c82361cc --- /dev/null +++ b/plugins/ActivitySpam/train.php @@ -0,0 +1,155 @@ +. + * + * @category Spam + * @package StatusNet + * @author Evan Prodromou + * @copyright 2012 StatusNet, Inc. + * @license http://www.fsf.org/licensing/licenses/agpl-3.0.html AGPL 3.0 + * @link http://status.net/ + */ + +if (!defined('STATUSNET')) { + // This check helps protect against security problems; + // your code file can't be executed directly from the web. + exit(1); +} + +/** + * Train a notice as spam + * + * @category Spam + * @package StatusNet + * @author Evan Prodromou + * @copyright 2012 StatusNet, Inc. + * @license http://www.fsf.org/licensing/licenses/agpl-3.0.html AGPL 3.0 + * @link http://status.net/ + */ + +class TrainAction extends Action +{ + protected $notice = null; + protected $filter = null; + protected $category = null; + + /** + * For initializing members of the class. + * + * @param array $argarray misc. arguments + * + * @return boolean true + */ + + function prepare($argarray) + { + parent::prepare($argarray); + + // User must be logged in. + + $user = common_current_user(); + + if (empty($user)) { + throw new ClientException(_("You must be logged in to train spam."), 403); + } + + // User must have the right to review spam + + if (!$user->hasRight(ActivitySpamPlugin::TRAINSPAM)) { + throw new ClientException(_('You cannot review spam on this site.'), 403); + } + + $id = $this->trimmed('notice'); + + $this->notice = Notice::staticGet('id', $id); + + if (empty($this->notice)) { + throw new ClientException(_("No such notice.")); + } + + $this->checkSessionToken(); + + $filter = null; + + Event::handle('GetSpamFilter', array(&$filter)); + + if (empty($filter)) { + throw new ServerException(_("No spam filter configured.")); + } + + $this->filter = $filter; + + $this->category = $this->trimmed('category'); + + if ($this->category !== SpamFilter::SPAM && + $this->category !== SpamFilter::HAM) + { + throw new ClientException(_("No such category.")); + } + + return true; + } + + /** + * Handler method + * + * @param array $argarray is ignored since it's now passed in in prepare() + * + * @return void + */ + + function handle($argarray=null) + { + // Train + + $this->filter->trainOnError($this->notice, $this->category); + + // Re-test + + $result = $this->filter->test($this->notice); + + // Update or insert + + $score = Spam_score::save($this->notice, $result); + + // Show new toggle form + + if ($this->category === SpamFilter::SPAM) { + $form = new TrainHamForm($this, $this->notice); + } else { + $form = new TrainSpamForm($this, $this->notice); + } + + if ($this->boolean('ajax')) { + $this->startHTML('text/xml;charset=utf-8'); + $this->elementStart('head'); + // TRANS: Page title for page on which favorite notices can be unfavourited. + $this->element('title', null, _('Disfavor favorite.')); + $this->elementEnd('head'); + $this->elementStart('body'); + $form->show(); + $this->elementEnd('body'); + $this->elementEnd('html'); + } else { + common_redirect(common_local_url('spam'), 303); + } + } +} diff --git a/plugins/ActivitySpam/trainhamform.php b/plugins/ActivitySpam/trainhamform.php new file mode 100644 index 0000000000..5a4c9c07af --- /dev/null +++ b/plugins/ActivitySpam/trainhamform.php @@ -0,0 +1,146 @@ +. + * + * @category Spam + * @package StatusNet + * @author Evan Prodromou + * @copyright 2011 StatusNet, Inc. + * @license http://www.fsf.org/licensing/licenses/agpl-3.0.html AGPL 3.0 + * @link http://status.net/ + */ + +if (!defined('STATUSNET')) { + // This check helps protect against security problems; + // your code file can't be executed directly from the web. + exit(1); +} + +/** + * Form + * + * @category Spam + * @package StatusNet + * @author Evan Prodromou + * @copyright 2011 StatusNet, Inc. + * @license http://www.fsf.org/licensing/licenses/agpl-3.0.html AGPL 3.0 + * @link http://status.net/ + */ + +class TrainHamForm extends Form { + + var $notice = null; + + function __construct($out, $notice) { + parent::__construct($out); + $this->notice = $notice; + } + + /** + * Name of the form + * + * Sub-classes should overload this with the name of their form. + * + * @return void + */ + + function formLegend() + { + return _("Train ham"); + } + + /** + * Visible or invisible data elements + * + * Display the form fields that make up the data of the form. + * Sub-classes should overload this to show their data. + * + * @return void + */ + + function formData() + { + $this->hidden('notice', $this->notice->id); + } + + /** + * Buttons for form actions + * + * Submit and cancel buttons (or whatever) + * Sub-classes should overload this to show their own buttons. + * + * @return void + */ + + function formActions() + { + $this->submit('train-ham-submit-' . $this->notice->id, + _('Clear spam'), + 'submit', + null, + _("Clear spam")); + } + + /** + * ID of the form + * + * Should be unique on the page. Sub-classes should overload this + * to show their own IDs. + * + * @return int ID of the form + */ + + function id() + { + return 'train-ham-' . $this->notice->id; + } + + /** + * Action of the form. + * + * URL to post to. Should be overloaded by subclasses to give + * somewhere to post to. + * + * @return string URL to post to + */ + + function action() + { + return common_local_url('train', array('category' => 'ham')); + } + + /** + * Class of the form. May include space-separated list of multiple classes. + * + * If 'ajax' is included, the form will automatically be submitted with + * an 'ajax=1' parameter added, and the resulting form or error message + * will replace the form after submission. + * + * It's up to you to make sure that the target action supports this! + * + * @return string the form's class + */ + + function formClass() + { + return 'form-train-ham ajax'; + } +} diff --git a/plugins/ActivitySpam/trainspamform.php b/plugins/ActivitySpam/trainspamform.php new file mode 100644 index 0000000000..ee1ecd2a74 --- /dev/null +++ b/plugins/ActivitySpam/trainspamform.php @@ -0,0 +1,146 @@ +. + * + * @category Spam + * @package StatusNet + * @author Evan Prodromou + * @copyright 2011 StatusNet, Inc. + * @license http://www.fsf.org/licensing/licenses/agpl-3.0.html AGPL 3.0 + * @link http://status.net/ + */ + +if (!defined('STATUSNET')) { + // This check helps protect against security problems; + // your code file can't be executed directly from the web. + exit(1); +} + +/** + * Form + * + * @category Spam + * @package StatusNet + * @author Evan Prodromou + * @copyright 2011 StatusNet, Inc. + * @license http://www.fsf.org/licensing/licenses/agpl-3.0.html AGPL 3.0 + * @link http://status.net/ + */ + +class TrainSpamForm extends Form { + + var $notice = null; + + function __construct($out, $notice) { + parent::__construct($out); + $this->notice = $notice; + } + + /** + * Name of the form + * + * Sub-classes should overload this with the name of their form. + * + * @return void + */ + + function formLegend() + { + return _("Train spam"); + } + + /** + * Visible or invisible data elements + * + * Display the form fields that make up the data of the form. + * Sub-classes should overload this to show their data. + * + * @return void + */ + + function formData() + { + $this->hidden('notice', $this->notice->id); + } + + /** + * Buttons for form actions + * + * Submit and cancel buttons (or whatever) + * Sub-classes should overload this to show their own buttons. + * + * @return void + */ + + function formActions() + { + $this->submit('train-spam-submit-' . $this->notice->id, + _('Train spam'), + 'submit', + null, + _("Mark as spam")); + } + + /** + * ID of the form + * + * Should be unique on the page. Sub-classes should overload this + * to show their own IDs. + * + * @return int ID of the form + */ + + function id() + { + return 'train-spam-' . $this->notice->id; + } + + /** + * Action of the form. + * + * URL to post to. Should be overloaded by subclasses to give + * somewhere to post to. + * + * @return string URL to post to + */ + + function action() + { + return common_local_url('train', array('category' => 'spam')); + } + + /** + * Class of the form. May include space-separated list of multiple classes. + * + * If 'ajax' is included, the form will automatically be submitted with + * an 'ajax=1' parameter added, and the resulting form or error message + * will replace the form after submission. + * + * It's up to you to make sure that the target action supports this! + * + * @return string the form's class + */ + + function formClass() + { + return 'form-train-spam ajax'; + } +}