Clemens

introduction

Since 2019 Klaudia and I are running the Mastodon instance literatur.social. Back then, we knew that we do not have the time to maintain a Mastodon fork or even a customized installation. We already saw several other smaller instances being haunted by their custom Mastodon installations and every security release would involve a lot of stress getting the code base updated in time.

While the resource footprint of our instance has changed in the last years, the setup itself is still pretty much the same, simple (ansible-managed) docker-compose setup running the official docker image, with easy maintainability as the main design goal.

In summer 2023, we had the need to back-port several upstream changes for admin-webhooks for our automatic account approval system. Additionally we wanted do a few smaller changes, like extending the number of possible poll options was something that we really wanted to try out and also a few (hard-coded) rate limits needed changes. A customized docker image was born.

managing and maintaining small changes with minimal effort

We don't want to maintain a full fork of the Mastodon repository. We basically just want to create and maintain simple, minimalistic patch files. One can do this with the basic diff and patch tools, but especially updating patches that break due to changes is not possible in a meaningful way.

This is where quilt comes in:

Quilt is a tool to manage large sets of patches by keeping track of the changes each patch makes. Patches can be applied, un-applied, refreshed, etc.

It is extensively used, among others, by OpenWrt to manage several hundred Linux kernel patches, by Debian to maintain patches for building packages and others. Both OpenWrt and Debian have quilt pages, but they are quite specific and you will probably find better how-tos.

a quick quilt introduction

For quilt you will have a patches folder with: – the patches/*.patch files you work on – a patches/series file that lists the currently enabled patches with their order

If you have worked with version control systems like git, working with quilt will feel quite familiar, although there are a few differences: – patches are created before making the actual changes – files are added/assigned to patches before making the changes – everything outside of the added files will not be tracked

A typical workflow looks like this: 1. quilt new 001-my-patch-change-foobar-patch – create a new patch 2. adding the file and making the changes: – quilt edit foo/bar/fileToChange.bar – directly edit with quilt and your favourite EDITOR, or – quilt add foo/bar/fileToChange.bar – add file to edit it in an external editor afterwards 3. quilt refresh – update the patch once you made your changes

The commands quilt push and quilt pop will allow you to apply and remove patches in your current working tree, changes are typically added to the last applied patch.

Disclaimer: This is not intended as a quilt tutorial, there is a lot more to it, (dealing with multiple patches, refreshing of failing patches, etc.) If more people are interested in this I might write another blog post, just write me a note at @datacop@literatur.social 😉

building a customized image

Building a custom Mastodon Docker image from scratch with the original Dockerfile would be the obvious way, but part from the build time and required build resources, the official image is well-tested and verified by the community within minutes of a new release.

This is why we decided to go the minimalistic way and apply our patches on top of the official release image by installing quilt, pushing the patches and re-compiling the assets.

This is the Dockerfile that we use to create our customized image:

FROM ghcr.io/mastodon/mastodon:latest
ARG DEVELOPMENT

# add a nice '+custom' version suffix
ENV MASTODON_VERSION_METADATA="custom"

USER root
RUN apt-get -y --no-install-recommends install quilt vim

# add patches and .quiltrc
COPY --chown=mastodon:mastodon patches /opt/mastodon/patches
COPY --chown=mastodon:mastodon .quiltrc /opt/mastodon/

# back to the mastodon user
USER mastodon
WORKDIR /opt/mastodon

# push all patches in the `series` file, do not fail if DEVELOPMENT
RUN quilt push -a -f || [ -n "${DEVELOPMENT}" ]

# Precompile assets again
RUN OTP_SECRET=precompile_placeholder SECRET_KEY_BASE=precompile_placeholder rails assets:precompile && \
    yarn cache clean

.quiltrc is just for nice looking patches and vim as default editor:

QUILT_DIFF_ARGS="--no-timestamps --no-index -p ab --color=auto"
QUILT_REFRESH_ARGS="--no-timestamps --no-index -p ab"
QUILT_PATCH_OPTS="--unified"
QUILT_DIFF_OPTS="-p"
EDITOR="vim"

docker build

To pull the lastest Mastodon release and build the image, run:

docker pull ghcr.io/mastodon/mastodon:latest
docker build . -f Dockerfile mastodon-custom

The image mastodon-custom will be your new, customized Mastodon image.

creating and updating patches

For creating and updating the patches I use this small shell script that builds a Mastodon container, starts it up with the local patches folder mounted as volume, and drops you in a shell where you can use quilt to do your changes and update your patches:

#!/bin/sh
set -xe
docker build . --build-arg DEVELOPMENT=1 -t mastodon-patch
docker run --rm -it -v $(pwd)/patches:/opt/mastodon/patches mastodon-custom /bin/bash
docker image rm mastodon-custom -f

conclusion

That's basically all the magic. This is an easy way to work with a few patches, and not lose the advantages of using the official Mastodon image. Especially backports of unreleased features from Mastodon's development branch were really easy to integrate.

We're currently running this in a GitLab instance to take advantage of GitLab's internal docker registry. On most Mastodon releases it is just needed to hit the rebuild button, once the Mastodon upstream image was updated.

Disclaimer: YMMV, further research is needed.

a word on dealing with major version updates

You might ask, But what if my patches break on updates?. Yes, you will have to fix and refresh your patches.

In our experience, smaller upstream changes are usually easily fixed by force-applying the patch, fixing every part of the patch that did not apply and refreshing the patch afterwards. We typically upgrade our instance within 1-2h of a new release.

For larger version updates, the upstream code might have changed so much, that you will have to re-write the patch. As soon as there are release-candidates, you can (and probably should) try to apply your patches on the release-candidate image tag to see if something breaks.

On bigger patch fails, find the pull request on GitHub that changed the parts you want to patch and understand what the upstream change intends to do. Mastodon's release notes are quite good, so the chances are quite high that you will find it there.