Yep. No contract means no authority. Said very well in 1870 by Lysander Spooner. http://praxeology.net/LS-NT-6.htm

The Constitution applies to those who have sworn to uphold and defend it. That includes members of the government, the military, the police, and naturalized citizens. It does NOT include the rest of us.