Ha! Yes indeed, I'm hearing you. The thing is, it might make someone feel 'uncomfortable' (a small step from awkward), and another person might decide that its inappropriate!
At this point, even though the comment is merely playful it is a small step to go from 'inappropriate' to taking offence.
Once someone takes offence the substance of the original intent no longer matters, the damage is done, and we just can't abide by this this now can we?

I hope this helps and, no offence, I didn't mean to be unclear ;)