The moral principles behind the First Amendment, behind the very fabric of Western civilization, demand not only that speech can't be censored in a healthy society; but also that speech can never be compelled in a healthy society.
To an extent, the tired phrase of "If you don't have anything nice to say, don't say anything at all." can be taken with certain wisdom. Of course, there need to be limits. Some people and ideas need to be told mean things. Sometimes, reality hurts.
What the "silence is violence" mantra does is destroy the possibility of wise discourse. Before "silence is violence", political correctness was simply an annoying system of lying. The authoritarians who believe in political correctness have always been that -- authoritarians. They've always demanded that you say what they want to hear or shut up. Still, they gave you the option to remain silent.
With "silence is violence", we have hoards of people who demand that you tell the lie with them lest you face the consequences. If you don't join in their chants, then you're deserving of whatever repercussions that the mob deems fit.
The irony of these people referring to themselves as anti-fascist shouldn't be lost on anyone.