A downvote is only as strong as their upvote.
I'd suggest not flagging anyone if you are afraid of that at the beginning.