This broke when changing the database to utf8mb4 instead of utf8,
since utf8_general_ci wasn't accepted and the engine fell back to
utf8mb4_bin. Now we're back in case insensitive search business!
Sometimes we just want to accept the user's wrong, but when it comes
to remote APIs etc. we probably want to let the client know it has
done something already (in this case multiple identical subscription
requests - which might indicate to it that it should refresh the sub
lists or something).
For example the public timeline would show notices from a user with
private_stream configured. (previously it would only hide _new_ notices
by this user as they would be the only ones with notice scoping set).
If a new file is uploaded, it will be matched with a previously uploaded
file so we don't have to store duplicates. SHA256 is random enough and
also unlikely enough to cause collisions.
If this merge throws exception on scripts/upgrade.php and you recently
tried a nightly (i.e. during 2015-02-19) then just go back a commit or two
and try again.
Or delete the duplicate entries. Find the entries like this:
SELECT COUNT(*), urlhash FROM file_redirection
GROUP BY urlhash
HAVING COUNT(*) > 1;
then for each urlhash (or come up with a smart SQL query) do:
DELETE FROM file_redirection WHERE urlhash='hashfrompreviousquery' LIMIT 1;
You'll have to remove duplicates more than once if you have >2 identical
urlhash entries. LIMIT -1 might do that for you. I'm not sure.
For some reason the "retweeted to me" part of the Twitter API was removed
when Evan made some inbox changes back in the StatusNet days. We might
recover this functionality, but not yet. The proper function calls are
however fixed in this commit.
Default is now to take still thumbnails of animated GIFs and then
show them as originals in an AttachmentListItem. The still frames
are mostly used with front-ends like qvitter.
The getUrl call would think that a File_thumbnail object was the child
of a local File if its filename was set. That has been true up to recent
development code where a File_thumbnail can have a 'filename' value,
but the original File does not. Only look at the File object to indicate
whether it's a local or remote file!
In some development code I noticed that when handling File objects without
filename values, there would be problems calling getPath and such.
The width and height value testing will be validated later anyway, and by
removing such a narrow test we can use events to generate thumbnails of
media formats supported by recently added plugins on demand.
This really should be a UUID or something else totally unexpected
but I figure that crc32 is good enough for now. The reason we keep
the main structure is because some third party scripts have begun
relying upon the tag URI format to parse out domain name, type etc.
This means we import the URI string from remote instances to track their
conversations and are able to stitch together replies in a single thread.
We might have to try to avoid collisions so noone remotely can predict
conversation URIs which we generate on our server, causing a DoS kind of
problem.
If we know the URI sent from the remote party, and we don't know the
notice it is replying to, we might still be able to put it in the same
conversation thread!
Getting rid of NoticeListItemAdapter, putting more into ActivityHandlerPlugin
and relying on plugins to handle rendering code of the content. This gives us
a lot more structure and consistency in notice structure and allows activity
plugins to stop rendering certain kinds of notices more easily.
There should also be a property for an ActivityHandlerPlugin class to avoid
rendering notices in the ordinary stream, so we don't have to overload stuff.
saveActivity will accept an Activity which gets parsed and saved through
plugins. So when an ActivityHandlerPlugin (such as Favorite will be soon)
gets a feed to save, this will be the function called instead of saveNew.
The code is now more event-driven when it comes to rendering notices
and their related HTML elements, since we can't have direct calls from
core to a plugin.
lib/activitymover.php has a function to move a Favorite activity which
will not happen now. The move must be pluginified and performed as an
event which plugins can catch on to.
Now we have to fix any code in the core which directly uses the Fave class
or any other favorite stuff, since it is pluginised and thus might not be
available on some installations.
No validation has been attempted yet. Lots of changes left. This
is visibly not (very) different from the previous CSS layout. But
some simplifications have been made.
Might cause issues with local changes to themes and CSS. Also maybe
javascript which depends on certain legacy microformats elements.
The move to microformats2 is motivated by the announcement that all
microformats should be migrated to version 2, as of 2014-06-20 at:
http://microformats.org/2014/06/20/microformats-org-turns-9-upgrade-to-microformats2
For a Notice object with multiple results, ->getProfile() would ALWAYS
return the first profile in the list. For example our "popular notices"
stream ended up believing all notices were made by the same profile.
We're now capable of doing image rotation for thumbnails based on
EXIF orientation data. Also, thumbnails are tracked by filenames and
thus we can delete them from storage when we feel like it.
Conversation trees works pretty bad with the current layout, javascript
etc. So it's best if we separate it and work on it as a side-project. The
oldschool settings are currently being deprecated (or broken out like this).
I'll wait with removing User preferences for oldschool conversation tree,
since that might be reusable data. But I guess it will go in the near future.
The File object now stores width and height of files that can
supply this kind of information. Formats which we can not read
natively in PHP do not currently benefit from this. However an
event hook will be introduced later.
The CreateFileImageThumbnail event is renamed to:
CreateFileImageThumbnailSource to clarify that the hooks should not
generate their own thumbnails but only the source image. Also it now
accepts File objects, not MediaFile objects.
The thumbnail generation is documented in the source code. For
developers, call 'getThumbnail' on a File object and hope for the best.
Default thumbnail sizes have increased to be more appealing.
The only reason it worked was because DB fetches calls to get$varname if
the dataobject has a variable with the specific name. However, it started
blurting out errors that the case must be correct (which would require
'geturl' to be the function name).
Since we probably want to replace DB sometime, we'll just override this
auto-fetching mechanism and use more explicitly defined functions.
Avoiding collisions with date (shorter than before) and 4 character
random alphanumeric string. I bet someone could mass-upload files
and generate all combinations of aaaa-zzzz during the course of a
day, but then maybe that user should be disabled anyway :)
(filling the collision space will cause a never-ending loop).
The exception thrown from MediaFile will be caught and simply result in
no thumbnail at all right now. In the future we might use a catch-all
and have a "cannot generate preview"-icon or something.
VideoThumbnails requires php5-ffmpeg and php5-gd.
Added the following FIXME:
How should a Twitter user get their Inbox filled with foreign tweets?
Every imported Twitter user has a profile in the Profile table, so we
could setup a Subscription entry for each of those, meaning they get
collected in the InboxNoticeStream... But this would mean a lot of
unnecessary entries and listings that generally just point to the
locked down Twitter service.
Let's figure out a good relation so we can connect any profile to any
imported foreign notice, so it shows up in the "all" feed.
This will work without much extra effort because there will always be
more notices (higher value) than conversations (so no collisions).
But please run upgrade.php to avoid having an autoincrement id on
conversation table.
Installations using code after 2014-03-01 will have identical
conversation IDs to the initial (conversation root) notice IDs. This
will not affect older installations, which will have very different
values.
Also removed the entirely unused saveGroups function.
Now avoiding multiGet and using listFind in Profile->getGroups()
so we don't have to deal with ArrayWrapper.
NoResultException was the wrong choice in this case, because it was
not a DB_DataObject instance that performed the search, but a static
call to the Notice class.
This reverts commit 8cc4660bd9.
This seems like something Evan only did to make pump.io import notices easier,
or maybe he just wanted to get rid of the identi.ca URLs?
This is the beginning of getting notice URI info via WebFinger
*XrdActionLinks is renamed *WebFingerProfileLinks, check EVENTS.txt
in WebFinger plugin for new events.
_flow_ reported on IRC that install.php had stopped working. This was
because default plugins had been put into two separate lists, and the
list with AuthCrypt was never loaded when performing an installation.
Core plugins cannot be disabled.
I also removed the Memcache autodetection thing since it should be
solved in a more elegant manner.
...no need to make a separate call to Local_group's setNickname all the time,
or a bunch of redundant code for the Profile table.
Next up is User->update()...
I implemented changes from quitter.se's new API that their front-end qvitter
uses, https://github.com/hannesmannerheim/qvitter/blob/master/api-changes-1.1.1/CHANGES
However I left out the URL shortening commens, since I believe whatever behaviour
they experienced that caused them to implement this was a bug (or many) and should
be fixed in their proper areas and that shortening should not be entirely left
out in API calls.
Profile_prefs aims to consolidate all the profile preferences into a
single table. Otherwise we end up with a bajillion *_prefs classes, like
User_urlshortener_prefs, or new fields in existing User/Profile classes,
like 'urlshorteningservice', 'homepage', 'phone_number', 'pet_name' etc.
Eventually we should migrate as many user-settable preferences as we can
into this system.
The data in Profile_prefs is organized by:
* profile_id Identify the current Profile.
* namespace Which plugin/section the preference is for.
* topic Preference name (like 'homepage')
* data Preference data (like 'https://gnu.org/')
The names 'topic' and 'data' are because 'key' and 'value' may be rather
ambigous when dealing with our DB_DataObject classes etc.
We're also now using $config['image']['jpegquality'] to determine the
quality setting for resized images.
To set Avatar max size, adjust $config['avatar']['maxsize']
The getAvatar call now throws exceptions too. Related changes applied.
Now let's move Profile->avatarUrl to the Avatar class!
* getOriginal added to Avatar class
This is a static function that retrieves the original avatar in a leaner
way than Profile->getOriginalAvatar() did (see below).
This will throw an Exception if there was none to be found.
* getProfileAvatars added to Avatar class
This gets all Avatars from a profile and returns them in an array.
* newSize added to Avatar class
This will scale an original avatar or throw an Exception (originally from
Avatar::getOriginal) if one wasn't found.
* deleteFromProfile added to Avatar class
Deletes all avatars for a Profile. This makes the code much smarter when
removing all avatars from a user.
Previously only specific, hardcoded (through constants) sizes would be
deleted. If you ever changed lib/framework.php then many oddsized avatars
would remain with the old method.
* Migrated Profile class to new Avatar::getOriginal support
Profile class now uses Avatar::getOriginal through its own
$this->getOriginalAvatar and thus remains backwards compatible.
* Updating stock GNU Social to use Avatar::getOriginal
All places where core StatusNet code used the
$profile->getOriginalAvatar, it will now useAvatar::getOriginal with
proper error handling.
* Updated Profile class to use Avatar::newSize
When doing setOriginal, the scaling will be done with the new method
introduced in this merge.
This also edits the _fillAvatar function to avoid adding NULL values to
the array (which causes errors when attempting to access array entries as
objects). See issue #3478 at http://status.net/open-source/issues/3478
New plugins:
* LRDD
LRDD implements client-side RFC6415 and RFC7033 resource descriptor
discovery procedures. I.e. LRDD, host-meta and WebFinger stuff.
OStatus and OpenID now depend on the LRDD plugin (XML_XRD).
* WebFinger
This plugin implements the server-side of RFC6415 and RFC7033. Note:
WebFinger technically doesn't handle XRD, but we serve both that and
JRD (JSON Resource Descriptor), depending on Accept header and one
ugly hack to check for old StatusNet installations.
WebFinger depends on LRDD.
We might make this even prettier by using Net_WebFinger, but it is not
currently RFC7033 compliant (no /.well-known/webfinger resource GETs).
Disabling the WebFinger plugin would effectively render your site non-
federated (which might be desired on a private site).
Disabling the LRDD plugin would make your site unable to do modern web
URI lookups (making life just a little bit harder).
maxContent==0 implies that a notice text can be infinitely long, but
this value was directly transferred to maxNoticeLength, where 0 was
tested if it was longer than the notice length - which of course always
was false.
This commit fixes the problem for infinite length notices that always
got shortened.
getUser calls are much more strict, and one place where this was found was
in the (un)subscribe start/end event handlers, which resulted in making the
Subscription class a bit stricter, regarding ::start and ::cancel at least.
Several minor fixes in many files were made due to this.
This does NOT touch the Foreign_link function, which should also have a more
strict getUser call. That is a future project.
Lots of the Memcached_DataObject classes stopped working when upgraded to
Managed_DataObject because they lacked schemaDef().
I have _hopefully_ made it so that all the references to the table uses
each class' schemaDef, rather than the more manual ColumnDef stuff. Not
all plugins have been tested thoroughly yet.
NOTE: This is applied with getKV calls instead of staticGet, as it was
important for PHP Strict Standards compliance to avoid calling the non-
static functions statically. (unfortunately DB and DB_DataObject still do
this within themselves...)
Not all instances of this has been fixed, but at least the ones
in the base class of Memcached_DataObject.
Avatar fix (in classes/Profile.php) requires a pkeyGet function
in the Avatar class (or as in this tree, the parent class of
Managed_DataObject)
Memcached_DataObject now defines
* pkeyGetClass to avoid collision with Managed_DataObject pkeyGet
* getClassKV to avoid collision with Managed_DataObject getKV
I used this hacky sed-command (run it from your GNU Social root, or change the first grep's path to where it actually lies) to do a rough fix on all ::staticGet calls and rename them to ::getKV
sed -i -s -e '/DataObject::staticGet/I!s/::staticGet/::getKV/Ig' $(grep -R ::staticGet `pwd`/* | grep -v -e '^extlib' | grep -v DataObject:: |grep -v "function staticGet"|cut -d: -f1 |sort |uniq)
If you're applying this, remember to change the Managed_DataObject and Memcached_DataObject function definitions of staticGet to getKV!
This might of course take some getting used to, or modification fo StatusNet plugins, but the result is that all the static calls (to staticGet) are now properly made without breaking PHP Strict Standards. Standards are there to be followed (and they caused some very bad confusion when used with get_called_class)
Reasonably any plugin or code that tests for the definition of 'GNUSOCIAL' or similar will take this change into consideration.
In some brief tests, this causes no problems.
In this state however, you would need to modify DB_DataObject to have a static declaration of staticget (and probably pkeyGet). The next commit will change the staticGet overload to a unique function name (like getKV for getKeyValue), which means we can properly call the function by PHP Strict Standards.
The parent class for our database objects, Managed_DataObject, has a
dynamically assigned class in staticGet which objects get put into,
leaving us with less code to do the same thing.
We will probably have to move away from the DB_DataObject 'staticGet'
call as it is nowadays deprecated.
We always call staticGet statically, so we define it statically. Next
step is to remove a bunch of definitions of 'staticGet' from classes
that can instead fall back to a parent class in Managed_DataObject.
The ampersand is removed as we're returning a class anyway, which does
not need a reference (and when we return false, it means nothing).
It may be a bad experience for new users to immediately when trying
out the service be asked for their geographical position. Instead,
let them opt-in for this behaviour.
If you look at classes/User_group.php on line 412 in the current code, you can see that a call to $profile->getGroups() is made. This implies getGroups($offset=0, $limit=PROFILES_PER_PAGE) only giving a limited amount of groups.
This means only the first 20 groups in an ascending numerical order by locally stored User_group->id will be addressable with the bangtag syntax.
I solved this by making the getGroups() call to the same one made in Profile->isMember(), i.e. $profile->getGroups(0, null);
User::getTaggedSubscriptions()
This change escapes the $tag argument to prevent a SQL injection
attack in User::getTaggedSubscriptions(). The parameter was not
escaped higher up the stack, so this vulnerability could be exploited.
This change escapes the argument to User::getTaggedSubscribers() to
prevent SQL injection attacks.
Both code paths up the stack fail to escape this parameter, so this is
a potential SQL injection attack.
This patch escapes query parameters in Profile_tag::getTagged(). This
is an extra security step; since these parameters come out of the
database, it's unlikely that they would have dangerous data in them.
This change adds additional escapes for arguments to
Profile_tag::moveTag(). The arguments are canonicalized in the API and
Web UI paths higher up the stack, but this change makes sure that no
other paths can introduce SQL injection errors.
This patch escapes the $tag parameter in
Profile::getTaggedSubscribers(). The parameter is not escaped either
in actions/subscriptions.php or in actions/apiuserfollowers.php. So
there is a potential for SQL injection here.
This change escapes a parameter in Local_group::setNickname(). Review
of the code paths that call this function sanitize the parameter
higher up the stack, but it's escaped here to prevent mistakes later.
Note that nickname parameters are normally alphanum strings, so
there's not much danger in double-escaping them.
This change escapes a parameter in Local_group::setNickname(). Review
of the code paths that call this function sanitize the parameter
higher up the stack, but it's escaped here to prevent mistakes later.
Note that nickname parameters are normally alphanum strings, so
there's not much danger in double-escaping them.
commit 74c5e4cce42ae601c07b447e100f097c15ebf9d2
Author: Evan Prodromou <evan@status.net>
Date: Thu Oct 20 12:48:52 2011 -0400
Add back in some optimization indices lost in schema conversion
commit ef5c2acfcd123b25910a1c8bb4ae01a3f9608e5e
Author: Evan Prodromou <evan@status.net>
Date: Thu Oct 20 12:29:57 2011 -0400
restore some of the lost optimized indices on notice table
commit fb1dfa9e98ded23fb5bdebae6465424a8cb8acd6
Author: Evan Prodromou <evan@status.net>
Date: Thu Oct 20 10:40:07 2011 -0400
Use popular notice stream for favorited page
commit e1d409ff738e39061ad35589d546ce9bed456975
Author: Evan Prodromou <evan@status.net>
Date: Thu Oct 20 10:32:23 2011 -0400
Use a caching stream for popular notice section
Instead of a big cached query, we now use a caching notice stream for
the popular notice section. It uses a single-table query at the
bottom, then scopes the notices and filters for silenced users. This
should be much nicer to our database servers.
Also clears the popular cache when someone favors or disfavors
something. A nice optimization would be to save the last weights and
re-calculate them at invalidation time, adding the new notice (or not)
depending on its own score. That will have to wait for another day,
though.
commit e9b7ab4c26c95e755adaff53c3957dcfca31c16b
Author: Evan Prodromou <evan@status.net>
Date: Thu Oct 20 10:31:14 2011 -0400
Let CachingNoticeStream users skip the ';last' optimization
This should resolve the issues darkip was reporting with user_im_prefs entries returning null immediately after insertion (seen with memcached off, so it was happening even with the built-in in-process cache in the Cache base class).
What was happening was that the initial pkeyGet() would end up saving a negative cache entry under the form with the fields sorted in the key, as via multicacheKey():
'statusnet:blaguette:user_im_prefs:screenname,transport:brionv,sms' => 'N;'
then we'd do an insert() on the new entry, saving cache entries for the non-sorted key names returned by _allCacheKeys():
'statusnet:blaguette:user_im_prefs:transport,screenname:sms,brionv' => 'O...'
'statusnet:blaguette:user_im_prefs:user_id,transport:1234,sms' => 'O...'
but the next query via pkeyGet() still saw the negative lookup cache from before, and came back with null.
Now, _allCacheKeys() sorts the fields in the keys by using the same key-builder function, and queries pick up the same thing you just inserted. :)
Memcached_DataObject doesn't quite fully understand unique indexes, and can't properly build cache keys for compound unique or primary keys.
Managed_DataObject has more information in its schema data, so we can build a proper list.
