The "flag" is basically a downvote or "unvote" as you say...these reasons are basically why it exists. Hope this helps :)