I was reading through the Swarm wiki and the section on banning a device or user. It seems to me that one way to avoid some of the difficulty is to use subjective moderation similar to the way Cabal does it. Essentially, each person is their own moderator. You can block any user you want at any time and simply never see their posts in the chat. Additionally, each user can designate a person to moderate for them as well. So in addition to not seeing chats from accounts you personally block, you also don’t see chats from those your designated moderators block.
Finally, and I think this is the most important step - the default moderator is appended to the invite. So someone joining a distributed chat (perhaps with some undesirable people in there) will only see chat messages from users who are not blocked by the default moderator.
Might this solve the issue of banning a user?