Commit Graph

62 Commits

Author SHA1 Message Date
Mikael Nordfeldth
fb4e9b234d Twitter Import improvements. Still buggy?
Apparently mrvdb has problems with duplicate inserts and missing files when
unlinking. It could be due to coding, or it could be due to parallelizing.
2013-10-04 13:36:45 +02:00
Mikael Nordfeldth
93e878d7ca Make better use of Subscription class
removed lib/subs.php as it was essentially only a wrapper for Subscription
2013-09-19 17:29:05 +02:00
Mikael Nordfeldth
de55d8f83b plugins onAutoload now only overloads if necessary (extlibs etc.)
lib/plugin.php now has a parent onAutoload function that finds most common
files that are used in plugins (actions, dataobjects, forms, libs etc.) if
they are put in the standardised directories ('actions', 'classes', 'forms',
'lib' and perhaps some others in the future).
2013-08-28 16:10:30 +02:00
Mikael Nordfeldth
2a4dc77a63 The overloaded DB_DataObject function staticGet is now called getKV
I used this hacky sed-command (run it from your GNU Social root, or change the first grep's path to where it actually lies) to do a rough fix on all ::staticGet calls and rename them to ::getKV

   sed -i -s -e '/DataObject::staticGet/I!s/::staticGet/::getKV/Ig' $(grep -R ::staticGet `pwd`/* | grep -v -e '^extlib' | grep -v DataObject:: |grep -v "function staticGet"|cut -d: -f1 |sort |uniq)

If you're applying this, remember to change the Managed_DataObject and Memcached_DataObject function definitions of staticGet to getKV!

This might of course take some getting used to, or modification fo StatusNet plugins, but the result is that all the static calls (to staticGet) are now properly made without breaking PHP Strict Standards. Standards are there to be followed (and they caused some very bad confusion when used with get_called_class)

Reasonably any plugin or code that tests for the definition of 'GNUSOCIAL' or similar will take this change into consideration.
2013-08-18 13:13:56 +02:00
Jean Baptiste Favre
f726c6b23f Makes TwitterBridge plugin use Twitter API 1.1 instead of API 1 which is deprecated and will be switched off soon 2013-02-18 23:22:45 +01:00
Jean Baptiste Favre
a10a34a3cf Make TwitterBridge plugin use Queueing system 2013-02-18 23:22:08 +01:00
Jean Baptiste Favre
dd97468c72 Add Twitter mention import support 2013-02-18 23:16:06 +01:00
Evan Prodromou
8cd1bdc686 Remove Jabber stuff from Twitter daemon 2012-07-06 10:46:48 -04:00
Brion Vibber
c169dcb522 Fixes for Twitter bridge breakage on 32-bit servers. New "Snowflake" 64-bit IDs have become too big to fit in the integer portion of double-precision floats, so to reliably use these IDs we need to pull the new string form now.
Machines with 64-bit PHP installation should have had no problems (except on Windows, where integers are still 32 bits)
2010-11-23 09:28:47 -08:00
Brion Vibber
62408fef09 Work in progress on twitter import daemon 2010-10-29 14:12:40 -07:00
Brion Vibber
d743539cf7 Fixups for twitter streaming daemon 2010-10-29 13:41:15 -07:00
Brion Vibber
47eada3a95 Work in progress on site streams-aware TwitterDaemon 2010-10-29 13:18:03 -07:00
Brion Vibber
e76028b629 Work in progress: starting on new TwitterDaemon using the Site Streams API -- code is incomplete, pulling bits from streamtest.php pending a chance to test the actual site-streams mode 2010-10-28 18:26:48 -07:00
Brion Vibber
0eaa26476c Split the guts of Twitter status -> notice import from twitterstatusfetcher daemon into TwitterImport class which can be called from other places, letting us reuse code for the streaming API. 2010-10-05 13:57:32 -07:00
Brion Vibber
1acc7d66c6 Always specify UTF-8 targt charset for html_entity_decode(); default is 8-bit ISO-8859-1 which causes things to break when we later pass them through things that expect to work with UTF-8. For instance, running through preg_replace() with the /u option results in NULL, leading to problems with OStatus and SubMirror generating their plaintext versions and doing length-cropping. 2010-09-30 11:29:31 -07:00
Evan Prodromou
78ed0348b0 Merge remote branch 'gitorious/0.9.x' into 0.9.x 2010-09-21 14:07:48 -04:00
Siebrand Mazeland
ba6538ed78 * i18n/L10n updates
* whitespace updates.
2010-09-18 17:45:18 +02:00
Evan Prodromou
398df867de use capturing parens for matching at-mentions in twitterstatusfetcher 2010-09-10 09:21:01 -04:00
Evan Prodromou
deb8f2a38b make our own twitter links if Twitter doesn't give us entities 2010-09-10 09:10:53 -04:00
Evan Prodromou
5c98dfd58c text substitution is Unicode-aware for twitterstatusfetcher 2010-09-07 15:54:06 -04:00
Evan Prodromou
d37f0da077 debug statements for setting/getting last id 2010-09-07 11:53:26 -04:00
Evan Prodromou
9793e4ceb4 Store foreign ID for synch info, not user ID 2010-09-07 10:53:11 -04:00
Evan Prodromou
21bea14140 remove data dump of retrieved statuses 2010-09-07 04:42:23 -04:00
Evan Prodromou
e53e152d6f use and save since_id parameter for home_timeline 2010-09-07 04:22:55 -04:00
Evan Prodromou
a46ba0b548 save Twitter mentions as StatusNet replies 2010-09-07 04:00:59 -04:00
Evan Prodromou
c2c6acbfd2 add back in # and @ for links 2010-09-07 04:00:59 -04:00
Evan Prodromou
2de6d0b03d add Twitter-approved links to Twitter statuses 2010-09-07 04:00:59 -04:00
Evan Prodromou
702d36257c truncate retweeted stuff if it's too long 2010-09-07 04:00:58 -04:00
Evan Prodromou
9847bffe99 correctly mark repeats from Twitter as non-local 2010-09-07 04:00:58 -04:00
Evan Prodromou
e20dc9fec3 correctly check for local notice to status mappings when notice originated here 2010-09-07 04:00:58 -04:00
Evan Prodromou
e9e06879bb use Notice_to_status to check for duplicates 2010-09-07 04:00:58 -04:00
Evan Prodromou
a1981770ce Save notice-to-status mapping in its own table
Introduce a table mapping notices to Twitter statuses. Initialize
this table at checkSchema() time. Save the mapping when we push
or pull statuses. Use the table to determine if a notice has a
Twitter equivalent.
2010-09-07 04:00:58 -04:00
Evan Prodromou
928239ecf8 do our own repeating so we can pass in a uri 2010-09-07 04:00:58 -04:00
Evan Prodromou
1bf4972cc0 remove require_once for disappeared TwitterBasicAuthClient 2010-09-07 04:00:57 -04:00
Evan Prodromou
23dfadd9a6 remove basic auth code for Twitter since it's no longer supported 2010-09-07 04:00:57 -04:00
Evan Prodromou
a8c90d0243 switch twitterstatusfetcher from friends_timeline to home_timeline 2010-09-07 04:00:57 -04:00
Evan Prodromou
1a54291cc7 debug code to dump new status data 2010-09-07 04:00:57 -04:00
Evan Prodromou
184ee7fc1f move inbox insert code to status saver main loop 2010-09-07 04:00:57 -04:00
Evan Prodromou
20f6ea4840 some info notices in twitter status fetch for repeats and replies 2010-09-07 04:00:57 -04:00
Evan Prodromou
5651f79466 if something's a retweet, save it as a repeat in bridge 2010-09-07 04:00:57 -04:00
Evan Prodromou
4d70ba9597 save reply status for Twitter notices 2010-09-07 04:00:57 -04:00
Evan Prodromou
ed99fd51e8 don't save statuses from protected users 2010-09-07 04:00:57 -04:00
Zach Copley
5b7b38ab66 Some fixes to make the twitterstatusfetcher behave better in a multi-site configuration 2010-03-27 23:43:05 +00:00
root
982edc653f Another typo 2010-03-04 23:30:36 -05:00
Zach Copley
72f72d17db - Fix cache handling in TwitterStatusFetcher
- Other stability fixes
2010-02-04 01:53:08 +00:00
Evan Prodromou
436b8c845e Merge branch '0.9.x' into inblob 2010-01-12 23:53:52 -08:00
Brion Vibber
ec145b73fc Major refactoring of queue handlers to support running multiple sites in one daemon.
Key changes:
* Initialization code moved from common.php to StatusNet class;
  can now switch configurations during runtime.
* As a consequence, configuration files must now be idempotent...
  Be careful with constant, function or class definitions.
* Control structure for daemons/QueueManager/QueueHandler has been refactored;
  the run loop is now managed by IoMaster run via scripts/queuedaemon.php
  IoManager subclasses are woken to handle socket input or polling, and may
  cover multiple sites.
* Plugins can implement notice queue handlers more easily by registering a
  QueueHandler class; no more need to add a daemon.

The new QueueDaemon runs from scripts/queuedaemon.php:

* This replaces most of the old *handler.php scripts; they've been refactored
  to the bare handler classes.
* Spawns multiple child processes to spread load; defaults to CPU count on
  Linux and Mac OS X systems, or override with --threads=N
* When multithreaded, child processes are automatically respawned on failure.
* Threads gracefully shut down and restart when passing a soft memory limit
  (defaults to 90% of memory_limit), limiting damage from memory leaks.
* Support for UDP-based monitoring: http://www.gitorious.org/snqmon

Rough control flow diagram:
QueueDaemon -> IoMaster -> IoManager
                           QueueManager [listen or poll] -> QueueHandler
                           XmppManager [ping & keepalive]
                           XmppConfirmManager [poll updates]

Todo:

* Respawning features not currently available running single-threaded.
* When running single-site, configuration changes aren't picked up.
* New sites or config changes affecting queue subscriptions are not yet
  handled without a daemon restart.
* SNMP monitoring output to integrate with general tools (nagios, ganglia)
* Convert XMPP confirmation message sends to use stomp queue instead of polling
* Convert xmppdaemon.php to IoManager?
* Convert Twitter status, friends import polling daemons to IoManager
* Clean up some error reporting and failure modes
* May need to adjust queue priorities for best perf in backlog/flood cases

Detailed code history available in my daemon-work branch:
http://www.gitorious.org/~brion/statusnet/brion-fixes/commits/daemon-work
2010-01-12 20:45:09 -08:00
Evan Prodromou
2db840b2f5 use Inbox in twitterstatusfetcher 2010-01-09 14:02:01 -08:00
Zach Copley
eb0a5977ff Don't try to broadcast the notice if something went wrong. 2009-12-04 20:39:50 +00:00
Zach Copley
cff12603cf Make imported Twitter notices show up via real time plugins. 2009-12-04 20:17:42 +00:00