Commit Graph

2463 Commits

Author SHA1 Message Date
ef0f65720e [StoreRemoteMedia] Remote images are now stored exactly on the necessary size for the thumb 2021-02-21 16:03:46 +00:00
22b5dd8567 [Media] Fix several issues
[StoreRemoteMedia] Upgrade plugin to use the new Media system

API Changes:
- Added getters to File to better formalize the ideas of the commit "[Media] Fix issues with database file storage"

UI Changes:
- Now presented thumbnails are actual thumbnails (bug fix)
- Attachment actions have a slightly more extended behaviour

Many other minor bug fixes...
2021-02-21 16:03:46 +00:00
e51520bd63 [Core] Add an event for StoreRemoteMedia and Embed 2021-02-21 16:03:46 +00:00
6028175bfc [Media] Fix issues with database file storage
Fixed file quota as well.

There can be more than one file for the same filehash IF the url are different.

Possible states:
  - A file with no url and with filename is a local file.
  - A file with an url but no filename is a remote file that wasn't fetched,
    not even the thumbnail.
  - A file with an url and filename is a fetched remote file (maybe just a
    thumbnail of it).
  - A file with no filename nor url is a redirect.

Routes:
  Given these states, updated routes so that an attachment can only be
  retrieved by id and a file by filehash.

Major API changes:
  File::getByHash now returns a yield of files

Major UI changes:
  - Now remote non stored files are presented.
  - /view became preferred
  - Redirects to remote originals are preferred.

Many other minor bug fixes...
2021-02-21 16:03:46 +00:00
Alexei Sorokin
06dfd91a82 Various fixes
Fix OAuth and Realtime issues introduced in 9a515b9234

[DATABASE] Fix an empty default value mistake introduced in
fde929b151

[DATABASE][PostgreSQL] Avoid use of pg_constraint.consrc, which was removed in
PostgreSQL 12.

[DATABASE][MariaDB] Fix a typo introduced in aed2344bd4

[DAEMON] Wrap an assignment inside "switch":
a follow-up to adc689cb15
2020-10-11 18:29:47 +03:00
Alexei Sorokin
4d8b04cda9 Clear out potential duplicates when semi-joining a union
Using a left outer join as a semi-join is not a valid approach.
Can still be used for an anti-semi-join.
2020-09-21 22:25:33 +03:00
Alexei Sorokin
aed2344bd4 Set the character set before making a connection
Ideally the character set should be set with the connection, and so this is
exactly what's being done now.

And now the character set code is attempted to be generalised.
2020-09-16 19:34:49 +03:00
Alexei Sorokin
52d67b0f44 Avoid ordering just by a timestamp
Try to also employ an id when possible.
Involves reworking some of the indices.
2020-09-15 16:59:27 +03:00
Alexei Sorokin
001629b6dd [Memcached_DataObject] Do not encache on insert
This resulted in N=0 and empty "modified" in cache.
2020-09-14 20:19:17 +03:00
Alexei Sorokin
b04469a252 [DATABASE] Make sure the session always uses UTF-8 and UTC 2020-09-12 15:40:39 +03:00
Alexei Sorokin
adc689cb15 Avoid use of assignments bare inside statements
Either use them in a subroutine call or put parentheses around the assignment.
2020-09-08 12:42:51 +03:00
Alexei Sorokin
d0f96a7023 [Profile] Extend the allowed length of nicknames to 191 characters 2020-09-06 21:20:24 +03:00
Alexei Sorokin
4884a97223 [Memcached_DataObject] Change how multiGet achieves an ordered result
The previous approach sent the key values twice, which for large sets is
twice as bad.

As an optional feature of this approach multiGet now allows retrieving tuples
in exact order and amount of the requested key values.
2020-09-03 18:11:12 +03:00
Diogo Cordeiro
817074a787 [ActivityPub] Fix DELETE 2020-08-29 20:32:18 +01:00
Alexei Sorokin
00c492891e [Foreign_link] Change the type of "credentials" to blob
TwitterOAuthClient::packToken uses \0 as a delimeter which can cause issues on
TEXT or VARCHAR.
2020-08-25 16:06:37 +03:00
Alexei Sorokin
b419c5cf7c [Queue_item] Let DataObject construct WHERE IN 2020-08-24 16:16:00 +03:00
Alexei Sorokin
b01974b665 [PostgreSQL] Set timezone to UTC
And remove some redundant neighbouring cruft:
utf8mb4 is already set in mysqlschema more than enough times.
2020-08-17 16:52:11 +03:00
Alexei Sorokin
27045f03f5 [SESSION] Session ID can be as long as 128 characters 2020-08-17 13:58:19 +03:00
Alexei Sorokin
071baf04fd [NOTICE] Store "url" as TEXT and not VARCHAR(191) 2020-08-17 13:35:16 +03:00
Alexei Sorokin
341e34b766 [DATABASE] Change collation handling
Before now table definitions could define collations only for MariaDB using the
MariaDB's collation names directly.
Now instead definitions get a slightly more abstract collation name syntax, but
only supporting the collations utf8mb4_bin and utf8mb4_unicode_(cs|ci) (wrapped
as utf8_bin, utf8_general_(cs|ci)), because those are the ones that have
practical use for GNU social.

Which also means that on MariaDB the formerly used utf8mb4_general_(cs|ci) have
been superseded by utf8mb4_unicode_(cs|ci), as they are the more modern
replacement.

Introduce collation support on PostgreSQL which results in use of the C (POSIX)
collation as utf8_bin and the und-x-icu collation as utf8_general_cs.
utf8_general_ci is also mapped to und-x-icu, which makes it case-sensitive,
unfortunately.
2020-08-16 23:41:28 +03:00
Alexei Sorokin
b65c200922 [File] Fix file deletion violating foreign keys 2020-08-13 23:56:31 +03:00
Alexei Sorokin
ac6510d481 [Memcached_DataObject] Thorough check of pivotGetClass arguments 2020-08-12 13:50:39 +03:00
Alexei Sorokin
99da7963d1 [Managed_DataObject] Remove the "timestamp" type
It is converted to "datetime" in lib/database/schema.php
2020-08-11 18:30:12 +03:00
Alexei Sorokin
15f7941daf [Profile] Allow grantRole to work when role exists 2020-08-11 15:00:57 +03:00
Alexei Sorokin
ce665baa88 [Queue_item] There is no "modified" attribute
Fixes a regression introduced in ec86de2bc4
2020-08-10 17:46:30 +03:00
Alexei Sorokin
f84dbb369f [DATABASE] Enable fulltext search by default
Also rename fulltext indices to more fitting names
and move the check from classes into database/schema.php
2020-08-08 18:08:06 +03:00
Alexei Sorokin
b20c0bdec7 Clean Notice_prefs and Fave_tally when a notice is deleted 2020-08-08 12:22:35 +03:00
Alexei Sorokin
e206995268 Set HTTP status codes with http_​response_​code() 2020-08-04 14:12:17 +03:00
Alexei Sorokin
92e8c40c55 [DATABASE] Add explicit indices for all foreign keys
This adds a requirement for all definitions that have foreign keys to also
require indices for all source (local) attributes mentioned in foreign keys.

MariaDB/MySQL creates indices for source attributes automatically, so this
serves as a way to get rid of those automatic indices and create clean explicit
ones instead.

In PostgreSQL, most of the time, indices on the source are necessary to
decrease performance penalty of foreign keys (like in MariaDB), but they aren't
created automatically, so this serves to remove that difference between
PostgreSQL and MariaDB.
2020-07-31 16:36:40 +03:00
Alexei Sorokin
0bfa747382 [DATABASE] Fix index identifiers and clean up redundant ones 2020-07-31 16:12:48 +03:00
Alexei Sorokin
ec86de2bc4 [DATABASE] Update "modified" in Managed_DataObject instead of a DBMS trigger
Instead of relying on the MariaDB's ON UPDATE CURRENT_TIMESTAMP trigger update
"modified" attributes in Managed_DataObject. Every raw query that needs
adjusting is adjusted, as they won't update "modified" automatically anymore.

The main goal behind this change is to fix "modified" updates on PostgreSQL.
2020-07-27 19:10:33 +03:00
Alexei Sorokin
341f3d0ea5 [DATABASE] Fix more incorrect uses of quotation in SQL 2020-07-26 15:28:05 +03:00
Alexei Sorokin
579120df70 [Notice] Fix clearReplies() and clearRepeats() 2020-07-26 15:12:00 +03:00
Alexei Sorokin
2861ae2823 [AuthCrypt] Password storage and comparison improvements
Password hashes are now stored in a TEXT attribute, not limited to 199 symbols.
That limitation makes no sense as password hashes are not the kind of
information to be indexed.

Actually replace crypt() with password_verify() for password checking, current
code left password_verify() unused.

Only update passwords when they use a different algorithm from the current
default. Previously "overwrite" meant rehashing every login.

Replace the "argon" boolean option with "algorithm" and "algorithm_options" for
better configurability.
The default remains whichever is default for PHP's password_hash.
2020-07-25 20:16:21 +03:00
Alexei Sorokin
33e9b57b78 [Profile][User_group] Fix profile deletion violating foreign keys 2020-07-24 13:09:02 +03:00
Alexei Sorokin
eefaf7a2b4 [DATABASE][Schema_version] Change the hashing algorithm to SHA3-512 2020-07-23 18:16:56 +03:00
Alexei Sorokin
7081720ecb [DATABASE] Check SQL boolean values with "IS TRUE"
This way UNKNOWN (NULL) explicitly turns to FALSE when three-valued logic is
reduced to binary.
In pgsqlschema, however, use "IS FALSE" as boolean attributes in pg_index are
non-nullable, there is no outer join and there's no clear preference for NULL
reduction.

Over-complicated constructions in TagCloud queries have been simplified, which
should not affect their performance.

Additionally, in TagCloud's lib/subscriptionspeopleselftagcloudsection.php
a typing mistake in an equi-join of "profile_tag" and "profile_list" on
"tagger" was fixed.
That regression was introduced in f446db8e2a
2020-07-21 18:06:39 +03:00
Alexei Sorokin
488bddb02a [NOTICE] Update index for verbs in ProfileNoticeStream
After adding a verb condition there, MariaDB now prefers the
("created", "id", "is_local") and ("profile_id", "verb", "created", "id")
indices for that query, even though they are slow for the job.
So replace them with ("is_local", "created", "id") and
("profile_id", "verb", "created", "id") respectively.
Also fix the naming of the ("profile_id", "created", "id") index.
2020-07-07 20:43:08 +03:00
Alexei Sorokin
f7d3f58318 [CORE] Do not set "confirm_address"."user_id" to 0 by default
That breaks the foreign key constraint, it is better to just have it nullable.
2020-07-06 20:01:16 +03:00
Diogo Cordeiro
fe4a9a6189 [ActivityPub][Ostatus] Fetch avatars in accordance to the new media system 2020-07-05 15:37:52 +01:00
Alexei Sorokin
9a515b9234 [SCHEMA] Improve timestamp storage
Avoid the use of deprecated MariaDB "zero dates" globally. If they're present
as attribute defaults somewhere, they will be replaced with NULL implicitly.
The existing "zero dates" in MariaDB storage will be left intact and this
should not present any issues.

The "timestamp" type in table definitions now corresponds to DATETIME in
MariaDB with "DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP", which
should be close enough to the original behaviour for compatibility purposes.
It is now the recommended type for "modified" attributes, because of the
update trigger on MariaDB. But there is no such trigger implemented on
PostgreSQL as of this moment.
2020-06-29 01:41:46 +03:00
Alexei Sorokin
b924c180ae [DATABASE] Fix MariaDB schema verification 2020-06-28 20:05:11 +03:00
Diogo Cordeiro
4544f29832 [AVATAR] Ensure this Action stays secure 2020-06-28 00:11:47 +01:00
Alexei Sorokin
edc7159ef6 [Memcached_DataObject] Check if it is possible to sort efficiently 2020-06-27 11:22:19 +03:00
Alexei Sorokin
0eec6fcfb6 [DATABASE] Replace NOW() with CURRENT_TIMESTAMP 2020-06-09 20:26:06 +03:00
Alexei Sorokin
6c035d01d4 [DATABASE] Start transactions with START TRANSACTION
"BEGIN" is non-standard and unnecessary.
2020-06-08 12:25:01 +03:00
Alexei Sorokin
e5ee069f4a [UI] Order groups in join descending order in profiles 2020-05-05 16:25:02 +03:00
Hugo Sales
d1c70cb13b [CORE] Make avatars be served with the same mechanism as attachments 2020-04-06 21:14:11 +00:00
Alexei Sorokin
e7ab305335 [CORE] Use monotonic time via hrtime() where applicable
The realtime clock is not reliable when calculating elapsed time.
2020-01-07 19:48:13 +03:00
Diogo Cordeiro
ca2dde9a41 [ExtendedProfile] Fix updates and allow to delete values 2019-11-03 19:49:10 +03:00