Free downvotes alone wouldn't really inspire me, I'm not really interested to move rewards from Bob with his mediocre content to Alice with her mediocre at least.
(And Alice might be Bobs girlfriend btw)
If on the other hand the downvoted rewards would be not just returned to the pool, but actually burned, that would be another story.