One vote drains 2% of your voting power, so if you vote less than 10x a day it will rise slowly. Or you wait 2,5 days (from 52%) until you are back to full steam. Concider that also the value of your votes declines with the voting power, so you upvote something with low VP but it has next to no value.