Euro guy here, so correct me if I'm wrong, but the way I understand your freedom of speech it only protects you from your government. I to doesn't give me the right to spout nonsense in a private space like your living room or on twitch and not get kicked out. You also have protected classes, but that's a different story.

Social repercussion, up to casting people out of communities, can actually be a highly valuable since very effective tool in such situations. Humans are social beings, after all. But you are right, it must be used sparingly, otherwise those outcast will form communities of their own and reinforce each other.