To me at least, my individual liberty stops when something I do infringes upon yours. There will always be rules against harm to others and their property, and I should also keep my word (contracts).

Differences in opinion should never be the deciding factor. The deciding factor should be whether someone or their property has been harmed or a contract they agreed to has been broken.