Most people naturally respond by saying “this is how we play this game. If you don't want to play, then leave us alone”. They never even think about the fact that they refuse to leave the libertarian alone, and in fact intend to disturb, harass, control and dominate him. In fact the very reality that they intend this makes the libertarian's position seem that much more bizarre to them.
They buy into the assumption that everyone is either jockeying to be the ones in control, or else they must be one of the subjects.
The unfortunate thing is that for most of human history, this has been a correct assessment.
People that are “playing the game”, who are jockeying for control, automatically assume that anyone who isn't in the game is one of the subjects to it.
They enjoy “playing the game”. It emboldens them. It makes them feel like they are a part of something. Even with their opposition (who they often are incredibly ruthless to) at least they know that they are all playing the same game together. They get even a small amount of unity out of that. They are so committed to this view of the world, that they assume that anyone who doesn't want to play must just be a subject. Its not even that they assume it, they positively insist on it.
The sad thing is that most people go along with it. Most people not only go along with it, they also depend on it. They see life as difficult. Challenging. Even overwhelming. No one is perfect and everyone ends up counting on other people at least a certain degree. Since most people acknowledge this, they end up taking a position as either the ruler or the ruled. The majority of the people end up behaving as the ruled.
The libertarian throws this entire dynamic out of whack. The libertarian insists that the game should not be played this way.
Most people have already built their internal identity somewhere on this scale between the ruled and the rulers. They accept it as the natural state of humanity and find their own niche within it.
When the libertarian rejects the game of power, most people don't even listen to the arguments. They just react emotionally. It really is received as a personal insult to their identity and their place in the world.
Most libertarians are familiar with this. It is normally a four step process. First people react with shock and incomprehension. Second they react with accusations like “how could that possibly work?”, “You need to prove to me how that would work”. Third they react with dismissal. They say “I have considered your position and found it lacking.” “How could you hold such a position?” “You are a bad person for thinking that”. Fourth they react with scorn. At this point they will do what they can to treat the libertarian as a bad person. A social outcast. They will try to make the libertarian feel guilty. They will try to make them compromise their beliefs in some way. But most importantly, they make make sure that everyone else knows that libertarian is the bad person, and the best thing that the libertarian can do is express regret, recant their position, confess their mistakes, conform to the groupthink, and submit. Now of course there are differences and variations, but this seems to be the trend.