My freedom to swing my fist doesn't really end at your nose, that's just a societal ideal. No force naturally springs up to interfere.
But it does my friend, it does; that force would be me :-) And then there'd be a fight and we'd have given up on all the good we could have achieved by joining our forces. And nature actually decided we need our forces joined in order to survive in the face of whatever the universe throws at us.
The universe foists upon us a great and terrible freedom, one that we scarcely dare consider or exercise.
Which is a good thing; we are social creatures after all. We are as free as however big a society allows us to be; absolute freedom is a pipe-dream or enjoyed by ourselves, removed from our fellow human beings.
Though we are profoundly limited by the possibilities presented by physics, cultures work to psychically limit the still incredible scope of what heroism or atrocities their members could commit.
The key word being "members", which implies we are indeed part of the greater whole that limits the atrocities at least...
Unless of course we have no free will, or everything is deterministic.
Well, as it looks now, free will is a neat illusion we need to make sense of reality of our place in the universe. But that's a whole other can of worms...
As always, your thoughts are highly appreciated, 
RE: True Freedom