I often find (at least for myself) that freelancing comes with a sort of in-built guilt: because of the lack of an imposed structure one might find in something like a corporate job, any bit of downtime makes you feel as if you're wrong to have it and you should be doing something, anything that'll advance your career or put more money in your account. And while, yes, being a successful freelance photographer means you have to grind away, you also run the risk of burning out, even though you're doing something you love.
I think it's important to give yourself permission to be tired and to take downtime. Recharging a bit will give you the energy and motivation to be at your best when the busy times strike again.