I personally think that the most fundamental reason for us to associate white with good thing and black with bad things is because of the darkness and light dichotomy. Darkness is scary and dangerous while light allows us to see what we are dealing with and allows us to feel safe. We are after all not nocturnal creatures. If we were, it would have probably been the other way around.

Still, this is a very primitive archetypal association and we by no means have to subscribe to it anymore. I personally prefer black to white in almost all kinds of scenarios - for instance, I wear black exclusively.

But extending our attitudes towards color (which are bound to be unreasonable as colors themselves have no intrinsic connotations) makes no sense and is obviously problematic.

To paraphrase what you said, people should be judged on their merits, not on their complexions.