You could ask the same question of peace keeping or any other sphere of public life where your freedoms are superseded by the rights of your neighbours. Obviously there would need to be an agreement and enforcement of that agreement, which I would call government. The question to libertarians is, how do you ensure that this authority's power comes from the bottom up, rather than the top down?