[DOCUMENTATION] Add documentation on installing without docker and other topics

docs/src/SUMMARY.md

@@ -5,4 +5,10 @@
     - [Instal without Docker with shell access](./install/no_docker_shell.md)
     - [Instal with Docker with web access](./install/docker_web.md)
     - [Instal without Docker with only web access](./install/no_docker_web.md)
+- [Configure]()
+    - [Queue](./queue.md)
+    - [Theme](./theme.md)
+    - [Private node](./private.md)
+- [Backups](./backups.md)
+- [Upgrading](./upgrading.md)
 

docs/src/backups.md

@@ -0,0 +1,6 @@
+# Backups
+
+There is no built-in system for doing backups in GNU social. You can make
+backups of a working StatusNet system by backing up the database and
+the Web directory. To backup the database use mysqldump <https://mariadb.com/kb/en/mariadb/mysqldump/>
+and to backup the Web directory, try tar.

docs/src/i18n.md

@@ -0,0 +1,14 @@
+### Internationalization and localization
+
+For info on helping with translations, see the platform currently in use
+for translations: https://www.transifex.com/projects/p/gnu-social/
+
+Translations use the gettext system <http://www.gnu.org/software/gettext/>.
+If you for some reason do not wish to sign up to the Transifex service,
+you can review the files in the "locale/" sub-directory of GNU social.
+Each plugin also has its own translation files.
+
+To get your own site to use all the translated languages, and you are
+tracking the git repo, you will need to install at least 'gettext' on
+your system and then run:
+    $ make translations

docs/src/install/bin-configure.md

@@ -0,0 +1,6 @@
+TODO more detail
+
+Run the `bin/configure` script and enter the information as asked.
+
+This will generate all the required `.env` files and (optionally) a
+`docker-compose.yaml` file.

docs/src/install/dns.md

@@ -0,0 +1,12 @@
+### Configuring DNS
+
+In order for your GNU social node to be accessible with your chosen
+hostname, you can create an `A` or `AAAA` DNS record, with your
+server's fixed IP v4 or v6 respectively in your DNS provider
+(normally, your domain registrar); the `A` record doesn't need to be
+at the root of your domain, meaning it's name can be a subdomain. For
+dynamic IPs, create a `CNAME` record pointing to the hostname you
+created with your chosen Dynamic DNS host. A `CNAME` cannot normally be created
+for a domain root, so you must use a subdomain. Note that some DNS
+providers provide 'CNAME flattening', in which case you can use your
+root domain.

docs/src/install/docker_shell.md

@@ -11,6 +11,10 @@ like to create containers for. This way you can use services in the
 host machine, which may be useful if your host already has a
 webserver, for instance.
 
+If you elect to not use some service containers, check [Instal without
+Docker with shell access](./install/no_docker_shell.md) for details on
+the configuration of each service.
+
 ## Prerequisites
 
 In order to host your GNU social instance, you'll need a domain:
@@ -25,69 +29,20 @@ or if you're behind a NAT, use a dynamic DNS solutions. Search for
 the `./install.sh` script. This allows you to have a domain that
 dynamically points to your IP address.
 
-## Configuring TLS/SSL
+{{#include dns.md}}
 
-You should configure a valid certificate and use TLS/SSL in most cases,
-one exception being wanting to use the Tor network.
-
-The `bin/configure` script is capable of setting this up for you, with
-the help of EFF's `certbot` and Let's Encrypt.
-
-There are multiple approaches to achieve this, among which are using
-your own (non-self) signed certificate, or using a proxy service
-capable of either proxying an HTTP connection to HTTPS (not
-recommended) or an HTTPS connection to HTTPS. For this approach,
-follow the instructions of your proxy service provider, but generally
-you'll use a self signed certificate, which the configuration script
-can generate.
-
-TODO Mail server configuration (links below)
-
-GNU social can be configured to send emails for various reasons. See
-[mail server configuration](). You'll need a certificate for your web
-domain and your mail domain, which may or may not be the same (if you
-use the same hostname for both, or a certificate valid for both).
-
-If you prefer to not use Let's Encrypt, pick `mixed` and uncheck the
-`certbot` service. Place your certificate in the folder
-`docker/certbot/.files/live/$HOSTNAME/`, where `$HOSTNAME` is the name
-where you want to host your node, such as `social.yourdomain`.
-Remember you also need a certificate for your mail server.
-
-TODO improve external certificate handling
-
-### Configuring DNS
-
-In order for your GNU social node to be accessible with your chosen
-hostname, you can create an `A` or `AAAA` DNS record, with your
-server's fixed IP v4 or v6 respectively in your DNS provider
-(normally, your domain registrar); the `A` record doesn't need to be
-at the root of your domain, meaning it's name can be a subdomain. For
-dynamic IPs, create a `CNAME` record pointing to the hostname you
-created with your chosen Dynamic DNS host. A `CNAME` cannot normally be created
-for a domain root, so you must use a subdomain. Note that some DNS
-providers provide 'CNAME flattening', in which case you can use your
-root domain.
-
-After this, run the `bin/configure` script (not as root).
-
-
-## Without TLS/SSL
-
-This is not recommended unless you know what you're doing. One
-exception is if you want your node to be used with the Tor network.
-
-Pick 'mixed' and uncheck the `certbot` service
-to disable it.
+{{#include tls.md}}
 
+{{#include no_tls.md}}
 
 ## Configuration
 
-TODO more detail
+{{#include bin-configure.md}}
 
-Run the `bin/configure` script and enter the information as asked.
-This will generate all the required `.env` files and (optionally) a
-`docker-compose.yaml` file.
+## Permissions
+
+The PHP docker container needs the GNU social folder to be owned by
+the group 82 (www-data).
 
 ## Running
 

docs/src/install/no_docker_shell.md

@@ -0,0 +1,197 @@
+# No Docker and shell installation
+
+## Prerequisites
+
+The following software packages are *required* for this software to
+run correctly.
+
+ - PHP 8.0+
+ - Postgres 10+/MariaDB 10.2+
+ - Web server
+ - Mail server
+
+Apache, lighttpd and nginx will all work. CGI mode is recommended and
+also some variant of 'suexec' (or a properly setup php-fpm pool)
+NOTE: mod_rewrite or its equivalent is extremely useful.
+
+The mail server is used for sending notifications and password resets,
+among other things.
+
+### PHP modules
+
+Your PHP installation must include the following PHP extensions for a
+functional setup of GNU social:
+
+ - bcmath     Arbitrary Precision Mathematics
+ - ctype      Locale support
+ - curl       Fetching files by HTTP.
+ - exif       Exchangeable image information.
+ - gd         Image manipulation (scaling).
+ - gmp        For Salmon signatures (part of OStatus)
+ - iconv      Locale support
+ - intl       Internationalization support (transliteration et al).
+ - json       For WebFinger lookups and more.
+ - mbstring   String manipulation
+ - mysql      The native driver for MariaDB connections.
+ - opcache    Improved PHP performance by precompilation
+ - openssl    (compiled in for Debian, enabled manually in Arch Linux)
+ - pcre       Perl Compatible Regular Expression
+ - readline   For interactive scripts
+ - Session    User sessions
+ - SimpleXML  XML parser
+ - Tokenizer  Reflection and annotations
+
+NOTE: Some distros require manual enabling in the relevant php.ini for
+some modules, even if they're included in the main PHP package.
+
+#### Better performance
+
+For some functionality, you will also need the following extensions:
+
+ - opcache       Improves performance a _lot_. Included in PHP, must be
+                 enabled manually in php.ini for most distributions. Find
+                 and set at least:  opcache.enable=1
+ - mailparse     Efficient parsing of email requires this extension.
+                 Submission by email or SMS-over-email uses this.
+ - sphinx        A client for the sphinx server, an alternative to MySQL
+                 or Postgresql fulltext search. You will also need a
+                 Sphinx server to serve the search queries.
+ - gettext       For multiple languages. Default on many PHP installs;
+                 will be emulated if not present.
+ - exif          For thumbnails to be properly oriented.
+
+You may also experience better performance from your site if you configure
+a PHP cache/accelerator. Most distributions come with "opcache" support.
+Enable it in your php.ini where it is documented together with its settings.
+
+{{#include dns.md}}
+
+{{#include tls.md}}
+
+{{#include no_tls.md}}
+
+### Getting it up and running
+
+Installing the basic GNU Social web component is relatively easy,
+especially if you've previously installed PHP packages.
+
+ 1. Download and unpack the release tarball or clone the `git` repository on
+    your Web server. Usually a command like this will work:
+
+    ```
+    tar zxf gnusocial-*.tar.gz
+    ```
+
+   ...which will make a `gnusocial-x.y.z` directory in your current directory.
+   (If you don't have shell access on your Web server, you may have to unpack
+   the tarball on your local computer and FTP the files to the server. Checkout
+   [Instal without Docker with only web access](./install/no_docker_web.md))
+
+ 2. Move the tarball to a directory of your choosing in your Web root
+    directory. Usually something like this will work:
+
+    ```
+    mv gnusocial-x.y.z /var/www/gnusocial
+    ```
+
+    This will often make your GNU social instance available in the gnusocial
+    path of your server, like "http://example.net/gnusocial". "social" or
+    "blog" might also be good path names. If you know how to configure
+    virtual hosts on your web server, you can try setting up
+    "http://social.example.net/" or the like.
+ 
+    You need "rewrite" support on your webserver. This is used for "Fancy URL"
+    support, which you can read more about further down in this
+    document.
+ 
+ 3. Make your target directory writeable by the Web server, please note however
+    that 'a+w' will give _all_ users write access and securing the webserver is
+    not within the scope of this document, but reading more on this subject is
+    recommended.
+
+    ```
+    chmod a+w /var/www/gnusocial/
+    ```
+ 
+    On some systems, this will work as a more secure alternative:
+
+    ```
+    chgrp www-data /var/www/gnusocial/
+    chmod g+w /var/www/gnusocial/
+    ```
+ 
+    If your Web server runs as another user besides "www-data", try
+    that user's default group instead. As a last resort, you can create
+    a new group like "gnusocial" and add the Web server's user to the group.
+ 
+ 4. Create a database to hold your site data. Something like this
+    should work (you will be prompted for your database password):
+
+    ```
+    mysqladmin -u "root" -p create social
+    ```
+ 
+    Note that GNU social should have its own database; you should not share
+    the database with another program. You can name it whatever you want,
+    though.
+  
+    (If you don't have shell access to your server, you may need to use
+    a tool like phpMyAdmin to create a database. Check your hosting
+    service's documentation for how to create a new database.)
+  
+ 5. Create a new database account that GNU social will use to access the
+    database. If you have shell access, this will probably work from the
+    MariaDB/PostgreSQL shell:
+  
+        GRANT ALL on social.*
+        TO 'social'@'localhost'
+        IDENTIFIED BY 'agoodpassword';
+  
+    You should change the user identifier 'social' and 'agoodpassword'
+    to your preferred new database username and password. You may want to
+    test logging in to MariaDB/PostgreSQL as this new user.
+  
+ 6. Run `bin/configure`
+
+{{#include bin-configure.md}}
+  
+ 7. You should now be able to navigate to your social site's main directory
+    and see the "Public Timeline", which will probably be empty. You can
+    now register new user, post some notices, edit your profile, etc.
+
+### Fancy URLs
+
+By default, GNU social will use URLs that include the main PHP program's
+name in them. For example, a user's home profile might be found at either
+of these URLS depending on the webserver's configuration and capabilities:
+
+    https://social.example.net/index.php/fred
+    https://social.example.net/index.php?p=fred
+
+It's possible to configure the software to use fancy URLs so it looks like
+this instead:
+
+    https://social.example.net/fred
+
+These "fancy URLs" are more readable and memorable for users. To use
+fancy URLs, you must either have Apache 2.x with .htaccess enabled and
+mod_rewrite enabled, -OR- know how to configure "url redirection" in
+your server (like lighttpd or nginx).
+
+TODO Add webserver sample configs
+
+1. See the instructions for each respective webserver software
+
+ - For Apache, inspect the `docs/webserver/htaccess.sample` file and save it as
+   `.htaccess` after making any necessary modifications. Our sample
+   file is well commented.
+ - For lighttpd, inspect the `docs/webserver/lighttpd.conf.example` file and apply the
+   appropriate changes in your virtualhost configuration for lighttpd.
+ - For nginx, inspect the `docs/webserver/nginx.conf.sample` file and apply the appropriate
+   changes.
+ - For other webservers, we gladly accept contributions of
+   server configuration examples.
+
+2. Ensure your webserver is properly configured and has its settings
+applied (remember to reload/restart it)
+

docs/src/install/no_tls.md

@@ -0,0 +1,7 @@
+## Without TLS/SSL
+
+This is not recommended unless you know what you're doing. One
+exception is if you want your node to be used with the Tor network.
+
+Pick 'mixed' and uncheck the `certbot` service
+to disable it, or `external`, if not using docker.

docs/src/install/tls.md

@@ -0,0 +1,33 @@
+## Configuring TLS/SSL
+
+You should configure a valid certificate and use TLS/SSL in most cases,
+one exception being wanting to use the Tor network.
+
+The `bin/configure` script is capable of setting this up for you if you use a
+Docker container. Otherwise, using [certbot](https://certbot.eff.org/) and
+[Let's Encrypt](https://letsencrypt.org/) is recommended
+
+There are multiple approaches to achieve this, among which are using
+your own (non-self) signed certificate, or using a proxy service
+capable of either proxying an HTTP connection to HTTPS (not
+recommended) or an HTTPS connection to HTTPS. For this approach,
+follow the instructions of your proxy service provider, but generally
+you'll use a self signed certificate, which the configuration script
+can generate.
+
+TODO Mail server configuration (links below)
+
+GNU social can be configured to send emails for various reasons. See
+[mail server configuration](). You'll need a certificate for your web
+domain and your mail domain, which may or may not be the same (if you
+use the same hostname for both, or a certificate valid for both).
+
+TODO improve external certificate handling
+
+If you prefer to not use Let's Encrypt, or the docker container, pick
+`mixed` and uncheck the `certbot` service or pick `external`.
+
+Place your certificate in the folder
+`docker/certbot/.files/live/$HOSTNAME/`, where `$HOSTNAME` is the name
+where you want to host your node, such as `social.yourdomain`.
+Remember you also need a certificate for your mail server.

docs/src/private.md

@@ -0,0 +1,30 @@
+### Private
+
+A GNU social node can be configured as "private", which means it will not
+federate with other nodes in the network. It is not a recommended method
+of using GNU social and we cannot at the current state of development
+guarantee that there are no leaks (what a public network sees as features,
+private sites will likely see as bugs).
+
+Private nodes are however an easy way to easily setup collaboration and
+image sharing within a workgroup or a smaller community where federation
+is not a desired feature. Also, it is possible to change this setting and
+instantly gain full federation features.
+
+Access to file attachments can also be restricted to logged-in users only:
+
+1. Add a directory outside the web root where your file uploads will be
+   stored. Use this command as an initial guideline to create it:
+
+       mkdir /var/www/gnusocial-files
+
+2. Make the file uploads directory writeable by the web server. An
+   insecure way to do this is (to do it properly, read up on UNIX file
+   permissions and configure your webserver accordingly):
+
+       chmod a+x /var/www/gnusocial-files
+
+3. Tell GNU social to use this directory for file uploads. Add a line
+   like this to your config.php:
+
+       $config['attachments']['dir'] = '/var/www/gnusocial-files';

docs/src/queue.md

@@ -0,0 +1,102 @@
+## Queues and daemons
+
+Some activities that GNU social needs to do, like broadcasting with OStatus or
+ActivityPub, SMS, XMPP messages and TwitterBridge operations, can be 'queued'
+and done by off-line bots instead.
+
+Run the queue handler with:
+
+```sh
+php bin/console messenger:consume async --limit=10 --memory-limit=128M --time-limit=3600
+```
+
+GNU social uses Symfony, therefore the [documentation on
+queues](https://symfony.com/doc/current/messenger.html#deploying-to-production)
+might be useful.
+
+TODO queuing
+
+#### OpportunisticQM plugin
+
+This plugin is enabled by default. It tries its best to do background
+jobs during regular HTTP requests, like API or HTML pages calls.
+
+Since queueing system is enabled by default, notices to be broadcasted
+will be stored, by default, into DB (table queue_item).
+
+Whenever it has time, OpportunisticQM will try to handle some of them.
+
+This is a good solution whether you:
+
+* have no access to command line (shared hosting)
+* do not want to deal with long-running PHP processes
+* run a low traffic GNU social instance
+
+In other case, you really should consider enabling the queuedaemon for
+performance reasons. Background daemons are necessary anyway if you wish
+to use the Instant Messaging features such as communicating via XMPP.
+
+#### Queue deamon
+
+It's recommended you use the deamon, you must be able to run
+long-running offline processes, either on your main Web server or on
+another server you control. (Your other server will still need all the
+above prerequisites, with the exception of Apache.) Installing on a
+separate server is probably a good idea for high-volume sites.
+
+1. You'll need the "CLI" (command-line interface) version of PHP
+   installed on whatever server you use.
+
+   Modern PHP versions in some operating systems have disabled functions
+   related to forking, which is required for daemons to operate. To make
+   this work, make sure that your php-cli config (/etc/php5/cli/php.ini)
+   does NOT have these functions listed under 'disable_functions':
+
+       * pcntl_fork, pcntl_wait, pcntl_wifexited, pcntl_wexitstatus,
+         pcntl_wifsignaled, pcntl_wtermsig
+
+   Other recommended settings for optimal performance are:
+       * mysqli.allow_persistent = On
+       * mysqli.reconnect = On
+
+2. If you're using a separate server for queues, install StatusNet
+   somewhere on the server. You don't need to worry about the
+   .htaccess file, but make sure that your config.php file is close
+   to, or identical to, your Web server's version.
+
+3. In your config.php files (on the server where you run the queue
+    daemon), set the following variable:
+
+       $config['queue']['daemon'] = true;
+
+   You may also want to look at the 'Queues and Daemons' section in
+   this file for more background processing options.
+
+4. On the queues server, run the command scripts/startdaemons.sh.
+
+This will run the queue handlers:
+
+* queuedaemon.php - polls for queued items for inbox processing and
+  pushing out to OStatus, SMS, XMPP, etc.
+* imdaemon.php - if an IM plugin is enabled (like XMPP)
+* other daemons, like TwitterBridge ones, that you may have enabled
+
+These daemons will automatically restart in most cases of failure
+including memory leaks (if a memory_limit is set), but may still die
+or behave oddly if they lose connections to the XMPP or queue servers.
+
+It may be a good idea to use a daemon-monitoring service, like 'monit',
+to check their status and keep them running.
+
+All the daemons write their process IDs (pids) to /var/run/ by
+default. This can be useful for starting, stopping, and monitoring the
+daemons. If you are running multiple sites on the same machine, it will
+be necessary to avoid collisions of these PID files by setting a site-
+specific directory in config.php:
+
+       $config['daemon']['piddir'] = __DIR__ . '/../run/';
+
+It is also possible to use a STOMP server instead of our kind of hacky
+home-grown DB-based queue solution. This is strongly recommended for
+best response time, especially when using XMPP.
+

docs/src/sms.md

@@ -0,0 +1,47 @@
+### SMS
+
+StatusNet supports a cheap-and-dirty system for sending update messages
+to mobile phones and for receiving updates from the mobile. Instead of
+sending through the SMS network itself, which is costly and requires
+buy-in from the wireless carriers, it simply piggybacks on the email
+gateways that many carriers provide to their customers. So, SMS
+configuration is essentially email configuration.
+
+Each user sends to a made-up email address, which they keep a secret.
+Incoming email that is "From" the user's SMS email address, and "To"
+the users' secret email address on the site's domain, will be
+converted to a notice and stored in the DB.
+
+For this to work, there *must* be a domain or sub-domain for which all
+(or most) incoming email can pass through the incoming mail filter.
+
+1. Run the SQL script carrier.sql in your StatusNet database. This will
+   usually work:
+
+       mysql -u "statusnetuser" --password="statusnetpassword" statusnet < db/carrier.sql
+
+   This will populate your database with a list of wireless carriers
+   that support email SMS gateways.
+
+2. Make sure the maildaemon.php file is executable:
+
+       chmod +x scripts/maildaemon.php
+
+   Note that "daemon" is kind of a misnomer here; the script is more
+   of a filter than a daemon.
+
+2. Edit /etc/aliases on your mail server and add the following line:
+
+       *: /path/to/statusnet/scripts/maildaemon.php
+
+3. Run whatever code you need to to update your aliases database. For
+   many mail servers (Postfix, Exim, Sendmail), this should work:
+
+       newaliases
+
+   You may need to restart your mail server for the new database to
+   take effect.
+
+4. Set the following in your config.php file:
+
+       $config['mail']['domain'] = 'yourdomain.example.net';

docs/src/theme.md

@@ -0,0 +1,23 @@
+# Themes
+
+As of right now, your ability change the theme is limited to CSS
+stylesheets and some image files; you can't change the HTML output,
+like adding or removing menu items, without the help of a plugin.
+
+You can choose a theme using the $config['site']['theme'] element in
+the config.php file. See below for details.
+
+You can add your own theme by making a sub-directory of the 'theme'
+subdirectory with the name of your theme. Each theme can have the
+following files:
+
+display.css: a CSS2 file for "default" styling for all browsers.
+logo.png: a logo image for the site.
+default-avatar-profile.png: a 96x96 pixel image to use as the avatar for
+users who don't upload their own.
+default-avatar-stream.png: Ditto, but 48x48. For streams of notices.
+default-avatar-mini.png: Ditto ditto, but 24x24. For subscriptions
+listing on profile pages.
+
+You may want to start by copying the files from the default theme to
+your own directory.

docs/src/upgrading.md

@@ -0,0 +1,5 @@
+# Upgrading
+
+Upgrading is strongly recommended to stay up to date with security fixes
+and new features. For instructions on how to upgrade GNU social code,
+please see the UPGRADE file.