Good point - beyond the actual impact of the downvote there is the impact on people's pride which can elicit a rather irrational response. One could ignore that - but hostile reactions do set and poison the tone on a platform.
So, yes, you might be right that downvotes are a bad idea even when their monetary impact is small.
RE: Opposing Perceptions, mismatched definitions, parity mismatch, voting, down voting, and "fairness"