That's why we need tools for it. It'd like to be able to mark users so that I'll see immediately if they have done something unwanted earlier.

But of course this is mostly a job for bots. Once a spammer is in the automatic downvote list, his account will be useless.