The amount of people who would be ruthless and criminal in a society where individual freedom was paramount would be very, very, very small. The vast majority of people are peaceful and capable of existing together. Their will always be conflict no matter what societal system is employed. A vast majority of the violence and hatred you see now is a product of the state. This fuels division in the population and props the current system up. If the state was gone most of the violence would leave with it. Communities would become established and people who were violent or aggressive would not be accepted by the communities and be outsiders. Eventually they would come around on their own to accept the ideas or have to adapt to being outsiders.