The term 'racism' gets used in a variety of ways by different people, and those different senses of the word get conflated (that is to say, people switch from one use to another without noticing). In one sense, it is a form of bigotry, i.e. to have beliefs or attitudes which are held onto irrationally in the face of contradicting evidence, out of fear, hatred, etc. In this sense, racism is irrational by definition, and therefore anything racist is clearly bad. All instances of racism should be repudiated, and no further argument is required to show this.
In another sense, it simply means discrimination on the basis of race. For instance, one might say that an immigration policy which, in effect, discriminates on the basis of race is racist. This sense of the word is distinct from the first. Such a policy may also happen to be motivated by irrational fears (or it may not), but simply pointing out that the policy is racist (in the sense of being discriminatory) is not sufficient to show that it is thus motivated. The word 'racist' is being used in a different sense. Thus, in order to show that the opponents reasons for supporting the policy are irrational (or even better, to show that the policy itself is wrong), further argument is required.
Similarly, the belief in non-superficial differences between races (or ethnic groups), such as differences in various capacities, or behavioural disposition, is often defined to be racist. Yet, such beliefs are not necessarily the product of bigotry, as I have defined it, i.e. holding onto beliefs or attitudes irrationally in the face of contradicting evidence. One may simply be mistaken (and would change their belief based on new evidence), or indeed the belief may be true (at least, a priori, you must concede). Yet such a belief, or the believer, is often called racist to imply bigotry. Again this doesn't automatically follow, and further argument is required to show that the belief is held irrationally (or even better, to show that the belief is false).
(The social justice definition of 'racism' is different still, but I will ignore it here.)
Now, I may be accused of defending racism. I do not defend racism in the sense of bigotry. Nor am I here defending any particular policy or belief about differences between races or ethnic groups. However, if we are to have an honest conversation about race, or about racism, we should avoid conflating our terms (and presuming those who merely disagree with us to be morally deficient).
Similar points could be made about sexism, homophobia, and so on.