I don't think I've had that experience, I really don't watch that closely. But I do from time to time have to reduce my 'Feed' just so I can comment effectively. I wouldn't take it too personally though, but I do understand your feelings too.
When a door closes a window opens... Something else will come along to fill the vaccuum!
Just keep doing what feels right for you... Good luck
RE: Being unfollowed by a whale/dolphin feels like shit