make the point that you're empowering one person to ruin everybody's fun.

Which is again the "This system can be abused, thus no one should use it!" thing.

If you are using the X-Card more than once, maybe twice in a game you should seriously think about it.

If you are actually uncomfortable about the topics you are either in the wrongest game for you (case: Spiders game, but I hate spiders!), and/or you should see a therapist (case: Everything is uncomfortable!).
The other players should tell you exactly this, and bid you farewell. If everyone is a cool human being, it will be alright with everyone, and the game can continue.

If you don't like the direction the plot is going you should leave. In this case you should not just "Use the X-Card" but also talk about it, because using the X-Card for saying "I don't like this direction" is a broader X-Card use.

X-Card player: "Spiders? Again? What is it with you and spiders? This is getting dull, what do the others think?"
Player A: "He's got a point …"
Player B: "Well, the plot is good though."
GM: (option A) "Okay, I guess … This time they're not spiders, but wolves …"/ (option B) "Oh, there is a reason for everything being spiders!"

If you don't want to express being bored by SPIDERS! SPIDERS! SPIDERS! people will assume you're uncomfortable with it though, making you leave the table.

If you want to push something on the players they don't want (be it by raising the X-Card or saying "I don't want that") you're a bad player. The GM also is a player, again.

If (one of) your player(s) is(/are) a bad audience, and can't cope with things you think are normal for your game someone should leave, though.