2022-11-20: The Sisyphean Effort of ActivityPub Migration

My ActivityPub/Fediverse instance runs Pleroma. For reasons, I'm less happy about this now than I was in 2018 when I stood it up. I'd really like to migrate to GotoSocial but moving between Fediverse software stacks tends to come with the wisdom of "Do it on a new domain, or it won't work in weird ways". I never liked this kind of folklore tech wisdom so I set out to figure out why it won't work. After enumerating the gotchas I became fairly convinced I could do it. I was wrong.

So what happens?

So you're running Mastodon, and you decide you want to try something else like GotoSocial, so the first thing you'd do is see if anyone else has made a tool to do this migration, or a doc, or something. No one has. There's an open issue for a migrator, but it probably won't be worked on for a long time. That's the state of every stack except Mastodon itself; Mastodon doesn't even have the open issue. You're on your own here.

So you decide hey, you don't care about keeping your posts, you'll just delete Mastodon, install GotoSocial (or Pleroma, or Misskey, or whatever) and re-follow everyone. Cool. Except you can't follow anyone and your posts don't seem to go out to other instances at all. You can't federate with anyone. What gives?

Every communication between instances occurs over an RSA signed HTTP transaction. The keys for these transactions sit in your instance's database, and there's a key for each user. The first time you talk to another instance, it caches your RSA public key and checks future signatures against this cached copy. If the key changes, the receiving instance rejects your messages and fetches. These cached keys seem to stick around forever, so if you lose your database, that user is just hosed for ever federating with anyone who has your key cached. In addition, the instance itself has a key and if that's lost, a lot of instance-level federation tasks stop working too.

In addition to this, users following you break. A "follow" is a bidirectional affair where both users have to record the event in their databases. Your instance has to record who followed you, the follower's instance records they followed you too. When you post, the message is federated to the instance of everyone who follows you, then that instance puts the message in follower timelines. Ergo both sides need to know about the follow link. If you blat your database, your instance forgets where to send posts.

There is no way to re-establish this link short of finding a way to get your followers to unfollow and refollow you.

These two things form the major barriers to just migrating between software stacks, or purging your database to start over.

The signing keys

The first problem's an easy one to solve. You simply need to migrate your signing keys between databases. Again, there's two: the key for your specific user, and an instance-wide key. The instance-wide key is assigned to an invisible user that the software does not expose in searches or timelines. What it's named depends on what software you're using.

In my case of trying to migrate Pleroma to GotoSocial I needed to export my keys, as well as the keys for internal.fetch, then import them into my user in GotoSocial and the user named after the domain.

Small fun little problem here though: Pleroma and Mastodon store keys in PKCS8 format, GotoSocial demands them in PKCS1 format. So you have to convert with some dark openssl magic.

In addition to this, GotoSocial's manner of storing these keys in the database is bonkers. I opted instead to create a JSON database dump and edit it before re-importing.

This part went off without a hitch though. I was able to get around the first major problem with just up and migrating between software in about 30 minutes from scratch.

The deep morass that is followers

The next problem was getting my following/follower links back in place so I didn't have to beg 200 people to unfollow and refollow me. I wasn't very concerned about people I followed; I could just script something to refollow them once the new instance was up. There's no way to force a remote user to follow you though.

The solution here is create entries in the new instance database to tell the instance these people already follow you. From a 1000 foot view this is easy. Pleroma and GotoSocial both represent follow links as simple "X follows Y" relationship. The problem is those relationships link to what must be very complete and specifically formatted in-database representations of entire ActivityPub identities.

This on its own isn't that bad. GotoSocial stores almost a one-to-one copy of the ActivityPub spec account data set you retrieve by requesting Accept: application/activity+json from a Mastodon instance. You can see this data with curl; if you want to try it on my old account on chitter.xyz you can do this: curl -H 'Accept: application/activity+json' https://chitter.xyz/users/trysdyn. If you have jq I recommend flexing it here to make that mess readable.

It only really took me about 10 minutes to have a filter that turns that into GotoSocial's JSON database dump representation of an account. There's not even really a concern about Pleroma formatting things differently and breaking stuff because GotoSocial currently doesn't federate correctly with Pleroma. Hah. However as I started running test cases on random instances of differing Mastodon versions, looping in Misskey, other GotoSocial instances, Honk... The inconsistencies started to pile up.

I could keep pounding on this and eventually write what'll turn into an AP Rosetta stone, but wow that's a lot of work and all I really want to do is bang out a nasty filter for a one-time use and forget about it.

The RSA key format difference cropped up here again, so I'd need to figure out what format keys are coming in and convert them. I'd also need to hash out weird edge cases like avatars (the "image" atom) sometimes not actually being images. I'm actually not sure what'd happen if someone managed to get an audio file stored as an avatar; probably bad things. Some software presents some fields others don't, some present the same data in different fields due to unclear points in the AP spec... It's a mess.

There's also the matter of ULIDs. GotoSocial uses a cousin of UUIDs for its identifier creation. It uses three different methods of generating these ULIDs, with no real rhyme or reason between them. ULIDs are ordered by creation time, and are intended for uses where this is important, so I imagine another thing I'd have to worry about is making sure the timestamps and ULIDs on accounts, follow links, and the like are ordered such that they mesh together properly. My Go isn't that good to figure out exactly where that comes into play.

The big worry about libraries

Around this point I was looking at having to somehow validate this nasty protocol converter I wrote actually produces correct user representations, as well as figuring out how to properly generate ULIDs, convert RSA keys from multiple incoming formats, handle instances that are in other codebases just missing certain fields, and testing the resulting dataset by verifying I can look up data on and message every single user that follows me (200+).

Frankly this was more than I wanted to do, but I think the thing that cemented the end of my patience was the realization that the ActivityPub library GotoSocial uses is likely either dead or in a deep maintenance-only mode.

I came to this discovery when I started researching an asinine and ill-advised workaround for the follower problem: shell games with Move events.

I like to move it move it

Mastodon was the first to implement the idea of account migrations. This is a feature that lets you flag an account as unused and redirect people to a new one instead. Anyone currently following you when you execute this will automatically follow you new account (if they have this function enabled) and your new account will automatically follow everyone your old account follows. It's intended to be used if you need to change instances, I had a much more awful use in mind for it.

On a technical level this is actually a bi-directional transaction. First you have to go to your destination account and have it begin to emit an atom called alsoKnownAs containing your old account's name. Then you execute the migration by federating out a Move event that instructs followers to re-point their follows.

Mastodon supports this in the UI, Pleroma supports it but you have to manually roll API calls to access it, GotoSocial does not support it at all. GotoSocial does not support it because the possibly-dead ActivityPub library it uses doesn't support it.

My plan though, before stubbing my toe on this, was to take my current Pleroma instance, perform an account move to another instance like chitter.xyz, set up GotoSocial and inject my RSA keys as noted above, then emit another Move to move back. This would, in theory anyway, force everyone to follow me again so GotoSocial could populate those users and links into the database without me having to do dangerous manual rolling of user objects.

Alas, GotoSocial cannot receive account migrations right now and it's possible it might not ever.

Admitting Defeat

At this point I'm fried. I wasted a good chunk of my weekend on this. The finish line is in sight if I just carried through with the account data converter and put it through some testing, but I don't have confidence in my knowledge of ActivityPub, Golang, and Mastodon's handling of weird federation edge cases to go through with it. Especially since the failure state for federations problems is "Your messages silently stop going anywhere"

It's not a total loss though; I leveled up my understanding of both Go and ActivityPub approximately ten-fold. I'm glad for that really, I haven't had an interesting project that I learned a lot from in awhile.

Don't know how I'm going to get off Pleroma now though. I might have to follow suit with the hundreds of attempted software migrations before me and just burn the domain and move to a new one. Or maybe I'll just do the migration, break the follower links, and try to get word out there to re-follow me.

tags: tech