You can't generalize such an argument. Of course it's not okay to physically harm someone JUST because their views are different from yours. In the case of Nazis their views are different and their views involve the violent extermination of other races and promotion of their own. Neo-Nazism is just a different point of view, it's an ideology that aims to spread its message of white supremacy.