You are limited by everyone else's liberty. It has nothing to do with anyone else's wishes or any agreements.

If you do something that violates someone else's equal and identical rights, you aren't exercising your liberty, you are just violating them like any common bully would do. Anyone doing this is subject to self defense- which may not be "smart" or safe, depending on the circumstances, but will never be wrong. You may choose not to defend yourself. You may be defended by someone else, or not. But none of that changes the reality of what liberty is or what your rights are.

"Enforcement" is a poor substitute for defense. You are right to defend yourself with violence (physical restraint or physical harm), if you so choose, as long as it is actually defense and not an initiation of force. The bad guys won't like this, obviously.

The Zero Aggression Principle doesn't depend on other people- it is a promise from me to you, telling you what to expect from me. You can accept it or ignore it.