I am not sure how much more simply I can put this - a discussion about rights can't even begin until the question of power imbalance is resolved. Who will make sure that your rights are recognized? If it is some minimalist state, can people elect not to participate in that state even if that state aspires to protect rights within a given territory?
RE: The "Social Contract" Excuse