[StoreRemoteMedia] Upgrade plugin to use the new Media system
API Changes:
- Added getters to File to better formalize the ideas of the commit "[Media] Fix issues with database file storage"
UI Changes:
- Now presented thumbnails are actual thumbnails (bug fix)
- Attachment actions have a slightly more extended behaviour
Many other minor bug fixes...
Fixed file quota as well.
There can be more than one file for the same filehash IF the url are different.
Possible states:
- A file with no url and with filename is a local file.
- A file with an url but no filename is a remote file that wasn't fetched,
not even the thumbnail.
- A file with an url and filename is a fetched remote file (maybe just a
thumbnail of it).
- A file with no filename nor url is a redirect.
Routes:
Given these states, updated routes so that an attachment can only be
retrieved by id and a file by filehash.
Major API changes:
File::getByHash now returns a yield of files
Major UI changes:
- Now remote non stored files are presented.
- /view became preferred
- Redirects to remote originals are preferred.
Many other minor bug fixes...
Fix OAuth and Realtime issues introduced in 9a515b9234
[DATABASE] Fix an empty default value mistake introduced in
fde929b151
[DATABASE][PostgreSQL] Avoid use of pg_constraint.consrc, which was removed in
PostgreSQL 12.
[DATABASE][MariaDB] Fix a typo introduced in aed2344bd4
[DAEMON] Wrap an assignment inside "switch":
a follow-up to adc689cb15
Ideally the character set should be set with the connection, and so this is
exactly what's being done now.
And now the character set code is attempted to be generalised.
"Magic quotes" were removed in PHP 5.4, no need to mitigate it anymore.
Avoid implode() with the join()-like order of arguments which was deprecated
since PHP 7.4 and implicitly since PHP 5.3.
Also avoid implode() with an implicit separator for stylistic reasons.
mktime() with no arguments has been deprecated since PHP 5.1.
ImageMagickPlugin:
- Remove animated thumbnail setting, we'll be able to use FFmpeg for performance
- Remove onFillImageFileMetadata and onCreateFileImageThumbnailSource(), these
are handled just fine by ImageFile
- Bump minor version number
README:
- Update
This check made registration impossible when welcomeuser didn't have validation
as well.
And rename the "grandfatherCutoff" option to "exemptBefore".
"Grandfathering" is a relatively obscure term linked to the history of the
United States of America, so replace that with something self-descriptive.
This adds a requirement for all definitions that have foreign keys to also
require indices for all source (local) attributes mentioned in foreign keys.
MariaDB/MySQL creates indices for source attributes automatically, so this
serves as a way to get rid of those automatic indices and create clean explicit
ones instead.
In PostgreSQL, most of the time, indices on the source are necessary to
decrease performance penalty of foreign keys (like in MariaDB), but they aren't
created automatically, so this serves to remove that difference between
PostgreSQL and MariaDB.
- getMessages() is now fetching from the Notice table as supposed
- every show{format}* method is properly updated to use Notice objects
- json and xml responses retrieve multi-recipients without compromising
backwards compatibility
Instead of relying on the MariaDB's ON UPDATE CURRENT_TIMESTAMP trigger update
"modified" attributes in Managed_DataObject. Every raw query that needs
adjusting is adjusted, as they won't update "modified" automatically anymore.
The main goal behind this change is to fix "modified" updates on PostgreSQL.
This way UNKNOWN (NULL) explicitly turns to FALSE when three-valued logic is
reduced to binary.
In pgsqlschema, however, use "IS FALSE" as boolean attributes in pg_index are
non-nullable, there is no outer join and there's no clear preference for NULL
reduction.
Over-complicated constructions in TagCloud queries have been simplified, which
should not affect their performance.
Additionally, in TagCloud's lib/subscriptionspeopleselftagcloudsection.php
a typing mistake in an equi-join of "profile_tag" and "profile_list" on
"tagger" was fixed.
That regression was introduced in f446db8e2a
Argument 3 passed to htmloutputter::input() must be of the type string or null, array given, called in /srv/gnusocial/plugins/Bookmark/forms/bookmark.php on line 166
Argument 1 passed to xmloutputter::text() must be of the type string, null given, called in /srv/gnusocial/plugins/ExtendedProfile/lib/extendedprofilewidget.php on line 556
There was no checking of attributedTo, actors and referent object IDs to make
sure they exist in the same domain. Therefore, one could spoof messages from
people by doing attributedTo: whoever-i-want-to-spoof
Avoid the use of deprecated MariaDB "zero dates" globally. If they're present
as attribute defaults somewhere, they will be replaced with NULL implicitly.
The existing "zero dates" in MariaDB storage will be left intact and this
should not present any issues.
The "timestamp" type in table definitions now corresponds to DATETIME in
MariaDB with "DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP", which
should be close enough to the original behaviour for compatibility purposes.
It is now the recommended type for "modified" attributes, because of the
update trigger on MariaDB. But there is no such trigger implemented on
PostgreSQL as of this moment.
Make common_sql_weight employ standard SQL functions for the timestamp
difference in seconds.
Also replace UTC_TIMESTAMP in the MariaDB-specific part with CURRENT_TIMESTAMP
as it is the only occurence and GNU social sets UTC as a default timezone.
In a delete_orphan_files.php script simplify the main query considerably.
In clean_profiles.php stop using COUNT as if it is ANY, that is unnecessary
punishment for the database. Instead implement the anti-join with a
left outer join.
In Autocomplete and Activitypub_profile use joins instead of a WHERE OR
anti-pattern for the semi-joins.
In lib/ui/galleryaction.php replace a CROSS JOIN with an INNER JOIN.
In actions/sup.php remove a redundant subquery: WHERE is applied before
grouping either way.
Activitypub_profile:
- Update do_insert to trigger TFN's assistance in inserting the profile
explorer:
- Use the new LRDD's method for grabbing profile aliases
Remove redundant setting retrival code.
An example config.php entry to activate the new config:
$config['site']['sensitivecontent']['hideforvisitors'] = true;
This should fix nulls on explorer lookups inputed by postman after generate_followers/getSubscribers, that I think were caused by calling common_profile_uri that, curiously, only handles local profiles
Use $object->sqlValue('NULL') (identical to DataObject_Cast'ing) instead and
fix related issues like (email|sms)settings considering these NULLs as a
false positive for the E-Mail address still being set when it's been removed.
There could also be security implications to the now-disabled approach of
considering 'NULL' strings as SQL NULLs.
ActivityPubPlugin:
- Rework onProfileDeleteRelated to account for the tables _rsa and _pending_follow_requests
- Update onEndShowAccountProfileBlock to stop creating the ap_profile if it doesn't exist (we'll handle this in a different manner)
Activitypub_profile:
- Remove unnecessary code from from_profile method and add return type information
Explorer:
- Update travel_collection to call itself instead of _lookup, that was wrong
Trying to enable the RedisCache with the latest nightly, getting this with the daemon:
sep 25 11:40:18 friedrich startdaemons.sh[21428]: PHP Fatal error: Uncaught Error: Call to a member function getPayload() on null in /var/www/social/plugins/RedisCache/RedisCachePlugin.php:96
sep 25 11:40:18 friedrich startdaemons.sh[21428]: Stack trace:
sep 25 11:40:18 friedrich startdaemons.sh[21428]: #0 /var/www/social/lib/util/event.php(89): RedisCachePlugin->onStartCacheSet('gnusocial:herds...', Object(HubSub), NULL, 86400, false)
sep 25 11:40:18 friedrich startdaemons.sh[21428]: #1 /var/www/social/lib/cache/cache.php(202): Event::handle('StartCacheSet', Array)
sep 25 11:40:18 friedrich startdaemons.sh[21428]: #2 /var/www/social/classes/Memcached_DataObject.php(520): Cache->set('gnusocial:herds...', Object(HubSub))
sep 25 11:40:18 friedrich startdaemons.sh[21428]: #3 /var/www/social/classes/Memcached_DataObject.php(52): Memcached_DataObject->encache()
sep 25 11:40:18 friedrich startdaemons.sh[21428]: #4 /var/www/social/classes/Managed_DataObject.php(50): Memcached_DataObject::getClassKV('HubSub', 'hashkey', 'a38b9dc516371af...')
sep 25 11:40:18 friedrich startdaemons.sh[21428]: #5 /var/www/social/plugins/OStatus/classes/HubSub.php(47): Managed_DataObject::getKV('hashkey', 'a38b9dc516371af...')
sep 25 11:40:18 friedrich startdaemons.sh[21428]: #6 /var/www/social/plugins/OStatus/lib/hubprepqueuehandler.php(68): HubSub::getByHashkey('https://herds.e...', 'https://raki.so...')
sep 25 11:40:18 friedrich startdaemons.sh[21428]: #7 /var/www/social/plugins/RedisQueue/classes/RedisQueueManager.php(58): HubPrepQueueHandl in /var/www/social/plugins/RedisCache/RedisCachePlugin.php on line 96
Sign in to j
Refactored Activitypub_profile::ensure_web_finger to Activitypub_profile::ensure_webfinger
Do not throw exceptions in the handling of this event because we don't
want to stop the regular search just because we were unable to find
ActivityPub actors or notes.
The code used to operate under the assumption that MariaDB doesn't support
quoting identifiers. Not only is that not exactly true, but MariaDB has
reserved keywords that cannot be used as table or column names unquoted.
Issue description as reported by aab:
About one day after enabling redis plugin:
PHP Fatal error: Uncaught Error: Call to a member function getPayload() on int in /var/www/html/plugins/RedisCache/RedisCachePlugin.php:96
Stack trace:
0 /var/www/html/lib/event.php(89): RedisCachePlugin->onStartCacheSet('gnusocial:gatea...', Object(Queue_item), NULL, 86400, false)
1 /var/www/html/lib/cache.php(202): Event::handle('StartCacheSet', Array)
2 /var/www/html/classes/Memcached_DataObject.php(496): Cache->set('gnusocial:gatea...', Object(Queue_item))
3 /var/www/html/classes/Memcached_DataObject.php(416): Memcached_DataObject->encache()
4 /var/www/html/classes/Managed_DataObject.php(620): Memcached_DataObject->update(Object(Queue_item))
5 /var/www/html/classes/Queue_item.php(74): Managed_DataObject->update(Object(Queue_item))
6 /var/www/html/lib/dbqueuemanager.php(75): Queue_item::top(Array, Array)
7 /var/www/html/lib/iomaster.php(287): DBQueueManager->poll()
8 /var/www/html/lib/iomaster.php(161): IoMaster->poll()
9 /var/www/html/scripts/queuedaemon.php(112): IoMaster->service()
10 /var/www/html/lib/spawni in /var/www/html/plugins/RedisCache/RedisCachePlugin.php on line 96
NoticeSearchAction:
- Add new event before finding query matches
ActivityPubPlugin:
- Subscribe new searchNotice event
- Bump minor version number
Activitypub_explorer:
- Update lookup to make remote-grabbing optional
ActivityPubPlugin:
- update grab_notice_from_url to make online grab optional
- subscribe events of user and profile deletion
- bump minor version number
Activitypub_inbox_handler:
- separate handle_delete for delete-note and delete-person
Activitypub_postman:
- add delete-person logic
Activitypub_delete:
- update validation method to check for the "Person" type
- update to_array method to target the activity
Both StartSubscribe and StartUnsubscribe had a wrong initial if-condition.
Furthermore, this events were calling Activitypub_profile::from_profile()
which is wrong because it creates the Activitypub_profile object when
the goal is only to check if it exists already.
OStatusPlugin:
- Stop adding the remote-follow button
- Subscribe to required RemoteFollow plugin events
- Drop main/ostatussub route and update urls to the main/RemoteFollowSub route
- Bump plugin minor version number
actions/ostatusgroup,
actions/ostatuspeopletag:
- Update urls to the main/RemoteFollowSub route
lib/util:
- Port required functions from OStatusSubAction and adapt to be used with the new events
lib/default.php
- Add RemoteFollow to the list of default plugins
RemoteFollowPlugin:
- Subscribe events to add the remote-follow button
RemoteFollowInitAction:
- Handles the remote-follow form and getting the redirection url for follow completion
RemoteFollowSubAction:
- Handles the remote profile pulling and actual following
ActivityPubPlugin:
- Subscribe DirectMessage events
Activitypub_inbox_handler:
- Update handle_create_note to create private messages
Activitypub_postman:
- Add create_direct_note for sending private messages
Activitypub_create:
- Update create_to_array to support the 'directMessage' attribute
- Add isPrivateNote to verify private activities
Activitypub_notice:
- Update create_note to support the 'directMessage' attribute
- Remove isPrivateNote
lib/models:
- Add Activitypub_message, the model in charge of private notes
Activitypub_profile:
- Fix subscription-counter getter functions, invalid profiles were being counted
apActorFollowingAction:
- Small rewrite of generate_following, didn't make sense to not use try-catch block
apActorFollowersAction:
- Small rewrite of generate_followers, didn't make sense to not use try-catch block
Note that this commit isn't intended to add support for sending such notes
in GS. Instead, we handle the reception, storage and direct reply to this
type of notices, in AP.
ActivityPubPlugin:
- Subscribe the event StartNoticeSave to hack answering non-public notes
Activitypub_create:
- Add 'directMessage' attribute to the Create activity, defaulting to false for now
- Update validation method: validate 'directMessage' and add debug
Activitypub_notice:
- Handle incoming unlisted/followers-only notes
- Add support for unlisted-replies
- Add method to verify private (direct) notices
inbox_handler:
- Add handler for CREATE Note
- Prepare logic for private-messaging
- Overall refactor: Class members were continuously being passed as function arguments without need
SharePlugin:
- Stop showing the announce button in non public posts
For reference (raised by rozzin in IRC):
* http://foldoc.org/module
* http://foldoc.org/library
* http://foldoc.org/plugin
As noted by XRevan86, modules are not necessarily non-essential.
As we will keep the modules directory in GS root [therefore, near to
plugins/], it is evidenced the difference between both.
This is a simple yet fundamental structural change. It doesn't change
functionality but makes clearer the way we understand GNU social's
internals.
This commit does the necessary rework to store private messages
as Notices and to support Federation. The plugin's README presents
some more detail about the changes and future work that is still
required to do.
ActivityPubPlugin:
- Change event-based notice distribution to queues logic
ActivityPub/lib:
- Add queue handler class activitypubqueuehandler.php
Misc:
- Add documentation for the (Start/End)InitializeQueueManager events
OStatusPlugin:
- Enqueue in the last position, as it should be. No need to worry about
the OMB comment, this protocol no longer have queue handlers that could
cause a conflict.
ActivityPubPlugin:
- Prevent sending a Delete for an Announce
Activitypub_announce:
- Update announce_to_array to add id, to and cc information to the retrieved object
Activitypub_follow:
- Add id to the arguments of follow_to_array, useful for Accept-Follow activities
Activitypub_notice:
- Fix notice validation, url isn't a MUST
Activitypub_inbox_handler:
- Make handle_follow use the received activity id for the later Accept-Follow
Activitypub_postman:
- Fix call to the updated announce_to_array
- Fix successive unnecessary calls to ActivityPubPlugin::actor_uri()
ActivityPubPlugin:
- Minor onDeleteOwnNotice rewrite
Activitypub_inbox_handler:
- Add deletion check to incoming notice
Activitypub_postman:
- Call the correct getUrl function
ActivityPubPlugin:
- Minor re-write of favor/disfavor event handlers
Activitypub_postman:
like/undo-like:
- fix proper getUrl() call
misc:
- make all activities accumulate errors (may be needed later) and log some information about it
ActivityPubPlugin:
- Fix of accepted activity verbs to include SHARES
- Add attention profiles to delivery when announcing
Activitypub_notice:
- New local function to retrieve original URL
- Removal of unnecessary 'Atom*' attributes
- Small fix to the ensuring of actor profile
Activitypub_profile:
- New local function to fetch AP profiles from a collection
Activitypub_postman:
- Fix url passed in the announce activity
Follow interaction:
- Fixed mini-bug where the subscriber profile was being used as the subscribed
- Updated cache subscription-related values in both instances
- Tested and working with local GS instances
Unfollow interaction:
- Updated cache subscription-related values in both instances
- Tested and working with local GS instances
Followers/Following collections:
- Now returning ActivityPub profiles only
- Stored collections in cache
Misc:
- Fix bug concerning the retrieval of public/private-key after in-function generation
This is not the same as the one in https://notabug.org/diogo/gnu-social-activitypub-plugin
Differences to the first "release"
-> Doesn't use guzzle nor has any composer dependencies
-> Supports HTTP Signatures
-> Has basic l10n/i18n
-> Some minor bug fixes
Seriously improved documentation
Now NodeInfo 2.0 is available at /api/nodeinfo/2.0.json
For active users we now also consider favourites and recently created accounts
Some further minor bug fixes and full review of the implementation
Advantages:
* Increases security by preventing direct access to file/
* We are careful and have a defined('GNUSOCIAL') || die() to prevent
direct access to GS files, but we may miss one or a vendor/extlib may
not be as careful
* Improves directory structure - It's more natural to physically
separate what is public from what are GNU social resources
DeletenoticeAction:
- Added tombstone check before deletion
NoticeListItem:
- Added tombstone check before showing delete-form
ActivityVerb:
- The plugin was overwriting the deletenotice route. Added stronger
regexp to the connected routes.
Router:
- Fix calls to connect, most of them were misusing the function's params
URLMapper:
- Minor fixes
- Documentation
- Add support for accept-header specification
Plugins/*:
- Fix calls to connect
This commit also fixes the translation in /plugins/OpenID/actions/finishopenidlogin.php#L203-L204 (s/Syncronize/Synchronize)
Sync is a bad technical jargon and we should use Synch instead.
Synch is already used in other parts of GNU social as seen in plugins/TwitterBridge/classes/Twitter_synch_status.php
UI:
- "Delete" area is now "Actions" area
- Updated themes to better reflect the changes
Routes:
- index.php?action=finishsyncopenid => finishsyncopenid
Translations:
- Updated OpenID translation files
- Updated OpenID POT file
Versioning:
- Bump OpenID minor version
- Bump GS patch version
Why would have labeling the Synchronize button of Sync been of bad taste? - answered by XRevan86:
In "synchronise" "ch" is a digraph meaning /k/ (actually /x/ turned into /k/ in English but whatever).
So… not separate letters.
It's like "ph" in "alphabet", or "sh" in "sheep", or "ch" in "chop" -- "ch" can mean a whole variety of sounds.
Original XMPPHP is no longer maintained
Therefore I've done some optimizations and imported some commits from birkner and zorn-v forks.
None of the forks really looked ready to be adopted...
Added two more options: delete image-only attachments; delete previews (like oembed thumbs)
Some further minor improvements.
Thanks to colegota for spotting this issue.
I noticed that each time a notice was accessed it'd do a remote lookup
with HEAD (and continue despite 404 etc.) and then another attempt to
download the resource. If this wasn't successful new attempts would be
made for each loading of the resource, which is extremely resource
intensive.
Whenever we can say "it's been n seconds since the last attempt" we
could probably enable this again - or just manually reload remote
thumbnails (as part of the StoreRemoteMedia plugin etc.)
WebSub is probably finalised before we make a release anyway. Here is
the official spec: https://www.w3.org/TR/websub/
Mostly just comments that have been changed. Some references to PuSH <0.4
are left because they actually refer to PuSH 0.3 and that's not WebSub...
The only actual code change that might affect anything is FeedSub->isPuSH()
but the only official plugin using that call was FeedPoller anyway...
This because some remote server might have used third party PuSH hubs
but switch and we don't know about it.
Possible risks here are of course MITM that could force us to rediscover
PuSH hubs from a feed they control, but that currently feels ... meh.
This feature filters users who may log in via CAS. This is useful when
both CAS and password authentication is enabled and there is a mismatch
between some GNU social account names and CAS user names. This prevents
CAS users from logging in as someone else on GNU social.
