Hey there, I totally understand how you're feeling, as I am a freelancer myself. While all points you made are valid, I'd like to add one of my own: vacations or holidays. I can never get away from home/work as much as I want to and that feeling that work is piling up and clients are waiting can totally ruin your "break".
I wouldn't change freelancing for anything in this world, though.
Regards and keep up the good work!