I think of downvotes like this...

1.) someone is breaking into a store to steal money that someone else earned. The store owner is inside and begins shooting. He may or may not kill (Lower to negative rep) the abuser. But it serves as a reminder of consequences.

2.) while an offender is breaking into the store, the community rises up and neutralizes their offender.

And lastly:
2.) just as a weapon can be used to defend, a bad person can use it to harass. Someone with 40k HP can destroy people's accounts with the click of a button. Someone with 15k HP can take the account of someone with 500HP to negative rep for any reason.

The last is a bad situation, obviously. Have people done it? Yes. But is it common, no. So why kneejerk because it happened a few times in a few years?