There have been reports that undelegations can cause the Voting Mana of the account that loses the delegation to drop significantly.
If you want to undelegate but you don't want to cause any major voting disruptions for the account which will lose the delegation, maybe you should wait until this issue will be resolved.
The issue has been filed on GitHub as #3008, for the steem repository:
https://github.com/steemit/steem/issues/3008
If you feel you can contribute with more details about your own experiences regarding this issue, you should provide them on the above GitHub issue. Or simply comment below and if there are important details, I'll make sure I'll add them to that raised issue for you.
If you have an alt account you don't use at all, you might try to test this by delegating a small amount of SP to it and then undelegating it, and document the entire process, especially the effect on the Voting Mana of the alt account. I would do it, but I need my alt operational for my actifit reports (I think I should create a new alt soonish, it seems handy).
That's it for today, but it seemed important to avoid even more potential user discomfort, at least not coming from my followers, unknowingly.