I have gotten this question, in different forms, over the years and it has a buried assumption built in. The problem is that it assumes that libertarianism is a binary proposition. It's all or none. A government or society is simply libertarian or it is not. I don't see how this would ever be true as most ideas seldom are expressed in idealistically pure form.
Libertarianism is a philosophy of non initiation of force. This happens all the time. It's happening right here on this forum. Any time people do not force each other to do something it's working. It only can be viewed from a perspective of 'purity' from an interaction level. So I think this is a flawed question.
Let's say we have a nation filled with libertarians that have fully accepted and try their best to live by the philosophy. There will always be people that decide to use force. They are not libertarian in action or philosophy so do you discredit the whole society now as 'not libertarian' or it 'does not work' because of this single failure?
Libertarianism is all around us. People participate in it all the time even if they are unaware of the philosophy or even if they do not accept it. Really the run-away idea of using force is the abnormal interaction. That is where failure lies. When we decide as individuals and groups in society that we must force people live by our demands. That is where it doesn't work.
We consider it unjust to steal even a $1 bill from our neighbor. Yet when government comes in play as an agent on our behalf then it suddenly becomes okay to take from others. By government action we give assumed authority over to do things we would never do as individuals because it creates an easy abstraction to behave in such a way. The "someone ought to do something" becomes a rationalization to let go of guilt and responsibility for something we supposedly care about and only empower our mechanism for blame when it is not done.
Not using force works all the time. It is when we force each other that the failures come into place and beget further failures when we don't recognize that is the core problem.