Server tech specs
Mementomori.social consists of many bits and pieces. This technical page is meant to tell you all about that.
Software
I'm running a fork based on latest Mastodon nightly from source, main branch. On top of the Mastodon features, we have bunch of our own. See Instance features.
Current optimizations
Puma is on its own server
PostgreSQL is on its own server
Database is on its own scalable SSD volume (storage for the next 100 years)
ElasticSearch is on its own server
Media uses fast Cloudflare R2 Zero Egress Distributed Object Storage (practically endless storage)
Assets delivered by brotli
Nginx optimized for resources
RAM optimized for services
Sidekiq splitted to services and optimized for plenty of enough RAM
More than enough resources for every server and service
Server infrastructure
This is where the magic happens. All Mementomori.social servers are running on Hetzner Virtual Private Server, located in Helsinki, Finland.
Puma server
The server specifications:
60 GB SSD volume for backups
On top of the regular Mastodon dependencies, the server software has
Redis and all the other cool Mastodon dependencies
PostgreSQL server
The server specifications:
40 GB SSD volume for PostgreSQL database, scalable up to 1TB
ElasticSearch server
The server specifications:
S3 Object storage for media
Media storage is provided by Cloudflare R2, Zero Egress Distributed Object Storage.
Tweaks
There are some extensive improvements to the default installation.
Sidekiq services and jobs
Sidekiq background jobs have been separated to 12 systemd services. I decided not to have them on a different server for now, didn't find the benefit in it yet.
mastodon@mementomori:~$ sudo ls /etc/systemd/system/ |grep mastodon-sidekiq
mastodon-sidekiq-1-default@.service
mastodon-sidekiq-1-ingress@.service
mastodon-sidekiq-2-default@.service
mastodon-sidekiq-2-ingress@.service
mastodon-sidekiq-default@.service
mastodon-sidekiq-ingress@.service
mastodon-sidekiq-mailers@.service
mastodon-sidekiq-pull@.service
mastodon-sidekiq-push@.service
mastodon-sidekiq-scheduler@.service
mastodon-sidekiq.service
mastodon-sidekiq-template.service
Files (you get the idea):
mastodon-sidekiq-1-default@.service
[Unit]
Description=mastodon-sidekiq-%j-queue
After=network.target
[Service]
Type=simple
User=mastodon
WorkingDirectory=/home/mastodon/live
Environment="RAILS_ENV=production"
Environment="DB_POOL=%i"
Environment="MALLOC_ARENA_MAX=2"
Environment="LD_PRELOAD=libjemalloc.so"
ExecStart=/home/mastodon/.rbenv/shims/bundle exec sidekiq -c %i -q %j
TimeoutSec=15
Restart=always
# Proc filesystem
ProcSubset=pid
ProtectProc=invisible
# Capabilities
CapabilityBoundingSet=
# Security
NoNewPrivileges=true
# Sandboxing
ProtectSystem=strict
PrivateTmp=true
PrivateDevices=true
PrivateUsers=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_INET
RestrictAddressFamilies=AF_INET6
RestrictAddressFamilies=AF_NETLINK
RestrictAddressFamilies=AF_UNIX
RestrictNamespaces=true
LockPersonality=true
RestrictRealtime=true
RestrictSUIDSGID=true
RemoveIPC=true
PrivateMounts=true
ProtectClock=true
# System Call Filtering
SystemCallArchitectures=native
SystemCallFilter=~@cpu-emulation @debug @keyring @ipc @mount @obsolete @privileged @setuid
SystemCallFilter=@chown
SystemCallFilter=pipe
SystemCallFilter=pipe2
ReadWritePaths=/home/mastodon/live
[Install]
WantedBy=multi-user.target
Crontabs
There are some periodically running jobs.
# This is required for vixie-cron (man cron)
# Check http://superuser.com/questions/264528/problem-with-random-in-crontab/264541#264541
SHELL=/bin/bash
# Auto renew Let’s Encrypt certs, two times a day at a random minute.
# https://gist.github.com/ahmedelgabri/cba569863cfed73eeee2614d28a02004
0 */12 * * * /etc/bin/certbot-renew.sh >/dev/null 2>&1
# Disk space monitor
*/10 * * * * /etc/bin/diskspace.sh >/dev/null 2>&1
#*/30 * * * * /etc/bin/diskspace-media.sh >/dev/null 2>&1
# Check rclone mount health
* * * * * /usr/bin/rclone-directory-check.sh >/dev/null 2>&1
# Backup to gdrive
0 3 * * * bash /usr/bin/backup.sh >/dev/null 2>&1
# Prune rclone logs every night
5 4 * * * rm /var/log/rclone/*.log* >/dev/null 2>&1
# Database heartbeat
* * * * * bash /etc/bin/check-db.sh >/dev/null 2>&1
# Database size heartbeat
*/3 * * * * bash /etc/bin/diskspace-db.sh >/dev/null 2>&1
# Streaming API check
* * * * * bash /etc/bin/check-streaming.sh >/dev/null 2>&1
# Mastodon related
5 0 * * * RAILS_ENV=production /home/mastodon/.rbenv/shims/bundle exec rake mastodon:media:clear
10 0 * * * RAILS_ENV=production /home/mastodon/.rbenv/shims/bundle exec rake mastodon:push:refresh
15 0 * * * RAILS_ENV=production /home/mastodon/.rbenv/shims/bundle exec rake mastodon:feeds:clear
# Fetch new users
0 */4 * * * cd /home/mastodon/suomalaiset-mastodon-kayttajat && php /home/mastodon/suomalaiset-mastodon-kayttajat/fetch.php > /dev/null 2>&1
# Prune mastodon stuff to save disk space
0 */12 * * * bash /etc/bin/mastodon-prune.sh >/dev/null 2>&1
# Index search results periodically
0 */4 * * * bash /etc/bin/mastodon-build-search-index.sh >/dev/null 2>&1
# Check sidekiq health
* * * * * /bin/bash -l -c 'cd /home/mastodon/live && bash /etc/bin/check-sidekiq.sh' >/dev/null 2>&1
# Build ElasticSearch index
0 0 * * * /bin/bash -l -c 'cd /home/mastodon/live && bash /etc/bin/mastodon-build-search-index.sh' >/dev/null 2>&1
Server status and monitoring
The admin gets a phone call from Better Uptime automation if anything mentioned on the status page goes down, by the minute.
Services status can be followed at status.mementomori.social. Status page, monitors and heart beats are provided by Better Stack.
Backups
Snapshots from Hetzner every night (full server backed up)
Cron scripts which backup everything to Hetzner Storage box and Google Drive (Enterprise, unlimited)
Backups from database and files separately
Manual backups each time anything is performed
