UserActivityStream -- used to create a full activity stream including subscriptions, favorites, notices, etc -- normally buffers everything into memory at once. This is infeasible for accounts with long histories of serious usage; it can take tens of seconds just to pull all records from the database, and working with them all in memory is very likely to hit resource limits.
This commit adds an alternate mode for this class which avoids pulling notices until during the actual output. Instead of pre-sorting and buffering all the notices, empty spaces between the other activities are filled in with notices as we're making output. This means more smaller queries spread out during operations, and less stuff kept in memory.
Callers (backupaccount action, and backupuser.php) which can stream their output pass an $outputMode param of UserActivityStream::OUTPUT_RAW, and during getString() it'll send straight to output as well as slurping the notices in this extra funky fashion.
Other callers will let it default to the OUTPUT_STRING mode, which keeps the previous behavior.
There should be a better way to do this, swapping out the stringer output for raw output more consitently.
Changes the replacement of Twitter "entities" from in-place reverse ordering ('to preserve indices') to a forward-facing append-in-chunks that pulls in both the text and link portions, and escapes them all.
This unfortunately means first *de*-escaping the < and > that Twitter helpfully adds for us.... and any literal &blah;s that get written. This seems to match Twitter's web UI, however horrid it is.
http://status.net/open-source/issues/2442
Notes:
* Mapstraction causes JavaScript errors in XHTML mode, breaking our code if we're run later so the link doesn't work to get back to Desktop.
* not 100% sure how safe feature detection is here?
* Currently will be useless but visible links if no JS available; need to fall back to server-side for limited browsers
file_quota is adjusted from the defined value to take into account the maximum upload size limits in PHP, or cropped to 0 if uploads are disabled.
This can be used by client apps to determine maximum size for an attachment.
Now using the original text form of @-mentions and #-tags, as in Twitter's own HTMLification.
Canonical forms are still used in generating links, where it's polite to match the canonical form.
Search highlighting was being done with a regex on raw HTML text, followed by a second regex undoing replacements within double-quoted attribute values.
This broke on imported Twitter messages, as the way we generate the markup uses single quotes on the attributes, which didn't get matched by the second regex.
I've replaced this do-then-undo cycle by dividing up the import HTML into freetext spans and tags; the freetext gets replaced, while the tags are left untouched.
Regressions caused by bad refactoring in commit 21feac3bea.
Test cases in tests/CommandInterpreterTest.php were made against the pre-refactoring code, and now check out with the fixed code.
Failures were caused by not changing logic structure when moving from multiple exit points (each if point would return directly with a null or an object) to setting a result variable and then falling through to a common exit point. Without the if statements being restructured, the result variable would just get overridden by the next case.
Workaround for deleted profiles still appearing in cached subscriptions/subscribers lists: if we couldn't fetch them, don't include them in the ArrayWrapper.
ArrayWrapper doesn't deal well with null entries, which aren't meant to happen in how it works. This code has recently changed from dying directly with a PHP fatal error in that case to throwing an exception, which allows tracking down the caller.
It looks like there might be some cases where profiles and their matching subscriptions get deleted, but the subscription entries don't get properly cleared from cache... that still bears further investigation. The regular code path looks ok; calls Subscription::cancel() from code called in Profile::delete(); but if they're batch-deleted instead of one row at a time, that could fail to trigger.
A repeat/retweet is roughly equivalent to an active direct post, so should follow the posting rules, rather than always sending over as we do for fave notifications.
Output from 0.9.6 PuSH feeds seems to have a rump <author> but no
<activity:actor>. It was overwriting valid and useful data set up at
subscribe time.
This fix tries to avoid overwriting data. However, it may prevent
updates that delete data.
Bug: 3028
Should fix issue #3027: twitter user avatars not getting imported.
Due to the change in URI, all twitter users that had been previously seen were getting new profile entries, which tried to save the same avatar. This would fail as Avatar.url has a unique index.
Note: now anything new seen in the last couple days in production will still potentially conflict.
Brion Vibber
Zach Copley
Evan Prodromou
Samantha Doherty