Mastodon upgrade workflow

This is the manual upgrade workflow for mementomori.social.

This document is outdated and being replaced by automation. See repo: mementomori-social/upgrade-workflow-automation.

Test on the development environment

Start with logging in as the right user on your local Linux development environment.

sudo su - mastodon

cd /opt/mastodon

This branch is 45 commits behind mastodon/mastodon:main.

Steps

1. Go to GitHub

2. Take note: "This branch is X commits behind mastodon/mastodon:main", add note to part above.

3. Sync fork with main

4. Fetch all changes:

git fetch --all

1. Checkout main, build and test:

yarn cache clean && git checkout main && git pull

# If needed, install whatever version is needed in this point
rbenv install 3.3.4

bundle install && yarn install && RAILS_ENV=production bundle exec rails assets:precompile

restart-mastodon

This is an alias for:

sudo systemctl restart mastodon-web mastodon-sidekiq mastodon-streaming && sleep 5 && sudo systemctl restart postgresql

1. Check migrations:

RAILS_ENV=production bundle exec rails db:migrate:status

1. Migrate:

RAILS_ENV=production bundle exec rails db:migrate

1. If needs manual migrations, add:

RAILS_ENV=production bundle exec rails db:migrate:up VERSION=20230724160715

1. After migrations you might need to clear cache before restart:

RAILS_ENV=production /opt/mastodon/bin/tootctl cache clear

1. Test:

• Does the audio notification work

• Does the emoji picker work

• Does everything seem normal

• Does the different feeds work (bookmarks, favs...)

• Does the toot edits work from the arrow https://mementomori.test/@rolle/111668884903746596

1. Reset search index (on dev first: sudo service elasticsearch start):

RAILS_ENV=production bin/tootctl search deploy --reset-chewy

1. Rebuild search index:

RAILS_ENV=production bin/tootctl search deploy --only accounts --concurrency 16 --batch_size 4096;
RAILS_ENV=production bin/tootctl search deploy --only statuses --concurrency 16 --batch_size 4096;

1. Restart all services:

restart-mastodon

1. Create new branch for new version based on fresh main/tag:

git branch mementomods-2024-07-11
git checkout mementomods-2024-07-11

1. Attempt to merge previous branch:

git merge mementomods-2024-07-04

1. If it doesn't work with merge: Check the mods from last working branch, apply them on top of the branch and push the new working branch.

2. Update version:

nano +70 -w .env.production

1. Recompile:

bundle install && yarn install && RAILS_ENV=production bundle exec rails assets:precompile

restart-mastodon

1. It's good to clear cache sometimes:

RAILS_ENV=production /opt/mastodon/bin/tootctl cache clear

1. Start Elasticsearch on dev:

sudo service elasticsearch start

1. Reset search index (on dev first: sudo service elasticsearch start):

RAILS_ENV=production bin/tootctl search deploy --reset-chewy

1. If needed, build index for search:

RAILS_ENV=production bin/tootctl search deploy --only accounts --concurrency 16 --batch_size 4096;
RAILS_ENV=production bin/tootctl search deploy --only statuses --concurrency 16 --batch_size 4096;

1. Restart:

restart-mastodon

1. It's now time to test (If you see "Oops!An unexpected error occurred.", it is completely okay because we don't have access to the outside world.)

2. When everything works, push changes to git:

git push

1. Set upstream when asked.

Install on production

1. When everything works properly, login to mementomori.social on mastodon user.

2. Backup first:

sudo bash /usr/bin/backup-db.sh

1. Fetch changes:

cd $HOME/live
git fetch --all
git checkout -b upstream/mementomods-2024-07-11
git pull upstream mementomods-2024-07-11

1. Check that all changes are there:

git log --oneline

1. Update version:

nano -w +62 .env.production

1. Rebuild:

yarn cache clean

# If needed, install whatever version is needed in this point
rbenv install 3.3.4

bundle install && yarn install && RAILS_ENV=production bundle exec rails assets:precompile

1. Migrate:

RAILS_ENV=production bundle exec rails db:migrate:status

RAILS_ENV=production bundle exec rails db:migrate

1. Clear cache:

RAILS_ENV=production /home/mastodon/live/bin/tootctl cache clear

1. Run immediately:

sudo su -
/usr/local/bin/restart-mastodon

1. Open debug log:

sudo journalctl -u mastodon-web.service -f

1. Or monitor fatals:

sudo journalctl -u mastodon-web.service -f |grep FATAL

1. Done!