The ActivityPub protocol itself has a bunch of drawbacks and limitations I really fail to grasp, even looking at it some years after it's been published. (Which was in 2018, and the fact that apparently there seems to be neither a formal process nor a working group to work towards update versions of that spec is an issue to tackle on its own). To me, the most pressing and confusing ones are:
Virtually everything is tied to fully-qualified HTTPS URIs, all along with the fact that (except for micro.blog) it's impossible to use a custom domain even on a non-self-hosted Fediverse instance. This is still a total WTF moment whenever I touch back on it. It seems weird start to end. In a situation where most users are on systems they aren't likely to run on their own (and subsequently don't own the domain the system uses), it's hard to impossible to really understand that approach. In the end it seems a commitment to certain actors who benefit from having huge servers and want to make sure that, despite all portability and own-your-data communications, users aren't easily able to “just” take their content and go elsewhere. There would have been prior art to do this in a smarter way, in Hubzilla, at the very least.
Lack of a proper scenario for backup/fallback accounts or nomadic identity. In an environment where everyone is encouraged to run a small, ideally single-user instance maybe on that Raspberry Pi living in ones own cupboard, services are likely to be less reliable than Twitter or Facebook. There is prior art how to solve this - in Hubzilla. In a sane world, this feature would have been essential, and indeed it's second to top of my list of gaps in the current ActivityPub
spec I fail to wrap my head around.
Lack of real data portability, even though this somehow goes along with the previous topics. There's no standardized way of migrating anything related to your account across random Fediverse platforms. The implementations that exist are proprietary to individual implementations and somehow interoperate here and there (like most platforms except Hubzilla seem to support contacts being imported from and exported to a common CSV style format), but everything else, specifically migrating posts, threads, communications, … do not just fail technically but also already conceptually - as importing big archives with myriads of messages all of a sudden might end up in clashing with the moderation policies on the server the account has just been moved to. At this point, certain completely unresolved core issues seem to shine through.
Lack of real privacy - in example by adding encryption rather than just message signature.
Lack of standardization of particular content types, see above. There are some approaches to that but it seems they're way too limited to be used (or just ignored by the implementers, arguably).
Lack of feedback on communication errors to the end user, see above. In some ways, it feels like ActivityPub is an HTTP based redesign of a lot of things initially designed in e-mail protocols, but without learning the lessons that could have been learnt from looking at 30+ years of e-mail in large-scale deployments.