# Server tech specs ### Software I'm running a fork based on latest Mastodon nightly from [source](https://github.com/ronilaukkarinen/mastodon), main branch. On top of the [Mastodon features](https://joinmastodon.org/), we have bunch of our own. See [Instance features](/mementomori.social/instance-features.md). ### Current optimizations * Puma is on its own server * PostgreSQL is on its own server * Database is on its own scalable SSD volume (storage for the next 100 years) * ElasticSearch is on its own server * Media uses fast Cloudflare R2 Zero Egress Distributed Object Storage (practically endless storage) * Assets delivered by brotli * Nginx optimized for resources * RAM optimized for services * Sidekiq splitted to services and optimized for plenty of enough RAM * More than enough resources for every server and service ## Server infrastructure This is where the magic happens. All Mementomori.social servers are running on [Hetzner](https://www.hetzner.com/) Virtual Private Server, located in Helsinki, Finland. ### Puma server The server specifications: * 8 vcpus * 32 GB RAM * 40 GB local disk * 60 GB SSD volume for backups On top of the regular Mastodon dependencies, the server software has * Latest [nginx](https://www.nginx.com/) * [Brotli](https://github.com/google/brotli) * [Redis](https://redis.io/) and all the other cool Mastodon dependencies ### PostgreSQL server The server specifications: * 8 vcpus * 32 GB RAM * 40 GB SSD volume for PostgreSQL database, scalable up to 1TB ### ElasticSearch server The server specifications: * 4 vcpus * 16 GB RAM * 80 GB local disk ### S3 Object storage for media Media storage is provided by [Cloudflare R2, Zero Egress Distributed Object Storage](https://www.cloudflare.com/products/r2/). ### Tweaks There are some extensive improvements to the default installation. ### Sidekiq services and jobs Sidekiq background jobs have been separated to 12 systemd services. [I decided not to have them on a different server for now](https://mementomori.social/@rolle/110366892496594617), didn't find the benefit in it yet. ```bash mastodon@mementomori:~$ sudo ls /etc/systemd/system/ |grep mastodon-sidekiq mastodon-sidekiq-1-default@.service mastodon-sidekiq-1-ingress@.service mastodon-sidekiq-2-default@.service mastodon-sidekiq-2-ingress@.service mastodon-sidekiq-default@.service mastodon-sidekiq-ingress@.service mastodon-sidekiq-mailers@.service mastodon-sidekiq-pull@.service mastodon-sidekiq-push@.service mastodon-sidekiq-scheduler@.service mastodon-sidekiq.service mastodon-sidekiq-template.service ``` Files (you get the idea): {% tabs %} {% tab title="" %} {% code title="" %} ```systemd [Unit] Description=mastodon-sidekiq-%j-queue After=network.target [Service] Type=simple User=mastodon WorkingDirectory=/home/mastodon/live Environment="RAILS_ENV=production" Environment="DB_POOL=%i" Environment="MALLOC_ARENA_MAX=2" Environment="LD_PRELOAD=libjemalloc.so" ExecStart=/home/mastodon/.rbenv/shims/bundle exec sidekiq -c %i -q %j TimeoutSec=15 Restart=always # Proc filesystem ProcSubset=pid ProtectProc=invisible # Capabilities CapabilityBoundingSet= # Security NoNewPrivileges=true # Sandboxing ProtectSystem=strict PrivateTmp=true PrivateDevices=true PrivateUsers=true ProtectHostname=true ProtectKernelLogs=true ProtectKernelModules=true ProtectKernelTunables=true ProtectControlGroups=true RestrictAddressFamilies=AF_INET RestrictAddressFamilies=AF_INET6 RestrictAddressFamilies=AF_NETLINK RestrictAddressFamilies=AF_UNIX RestrictNamespaces=true LockPersonality=true RestrictRealtime=true RestrictSUIDSGID=true RemoveIPC=true PrivateMounts=true ProtectClock=true # System Call Filtering SystemCallArchitectures=native SystemCallFilter=~@cpu-emulation @debug @keyring @ipc @mount @obsolete @privileged @setuid SystemCallFilter=@chown SystemCallFilter=pipe SystemCallFilter=pipe2 ReadWritePaths=/home/mastodon/live [Install] WantedBy=multi-user.target ``` {% endcode %} {% endtab %} {% endtabs %} ### Crontabs There are some periodically running jobs. {% tabs %} {% tab title="User: root" %} ```sh # This is required for vixie-cron (man cron) # Check http://superuser.com/questions/264528/problem-with-random-in-crontab/264541#264541 SHELL=/bin/bash # Auto renew Let’s Encrypt certs, two times a day at a random minute. # https://gist.github.com/ahmedelgabri/cba569863cfed73eeee2614d28a02004 0 */12 * * * /etc/bin/certbot-renew.sh >/dev/null 2>&1 # Disk space monitor */10 * * * * /etc/bin/diskspace.sh >/dev/null 2>&1 #*/30 * * * * /etc/bin/diskspace-media.sh >/dev/null 2>&1 # Check rclone mount health * * * * * /usr/bin/rclone-directory-check.sh >/dev/null 2>&1 # Backup to gdrive 0 3 * * * bash /usr/bin/backup.sh >/dev/null 2>&1 # Prune rclone logs every night 5 4 * * * rm /var/log/rclone/*.log* >/dev/null 2>&1 # Database heartbeat * * * * * bash /etc/bin/check-db.sh >/dev/null 2>&1 # Database size heartbeat */3 * * * * bash /etc/bin/diskspace-db.sh >/dev/null 2>&1 # Streaming API check * * * * * bash /etc/bin/check-streaming.sh >/dev/null 2>&1 ``` {% endtab %} {% tab title="User: mastodon" %} ```sh # Mastodon related 5 0 * * * RAILS_ENV=production /home/mastodon/.rbenv/shims/bundle exec rake mastodon:media:clear 10 0 * * * RAILS_ENV=production /home/mastodon/.rbenv/shims/bundle exec rake mastodon:push:refresh 15 0 * * * RAILS_ENV=production /home/mastodon/.rbenv/shims/bundle exec rake mastodon:feeds:clear # Fetch new users 0 */4 * * * cd /home/mastodon/suomalaiset-mastodon-kayttajat && php /home/mastodon/suomalaiset-mastodon-kayttajat/fetch.php > /dev/null 2>&1 # Prune mastodon stuff to save disk space 0 */12 * * * bash /etc/bin/mastodon-prune.sh >/dev/null 2>&1 # Index search results periodically 0 */4 * * * bash /etc/bin/mastodon-build-search-index.sh >/dev/null 2>&1 # Check sidekiq health * * * * * /bin/bash -l -c 'cd /home/mastodon/live && bash /etc/bin/check-sidekiq.sh' >/dev/null 2>&1 # Build ElasticSearch index 0 0 * * * /bin/bash -l -c 'cd /home/mastodon/live && bash /etc/bin/mastodon-build-search-index.sh' >/dev/null 2>&1 ``` {% endtab %} {% endtabs %} ### Server status and monitoring The admin gets a phone call from Better Uptime automation if anything mentioned on the status page goes down, by the minute. Services status can be followed at [status.mementomori.social](https://status.mementomori.social/). Status page, monitors and heart beats are provided by [Better Stack](https://betteruptime.com). ### Backups * Snapshots from Hetzner every night (full server backed up) * Cron scripts which backup everything to Hetzner Storage box and Google Drive (Enterprise, unlimited) * Backups from database and files separately * Manual backups each time anything is performed --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://help.mementomori.social/mementomori.social/server-tech-specs.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.