I changed on this position as well just like Gorski. I used to think a liberal order should generally allow religious/philosophical exemptions to vaccination. But they aren't really feasible if one intends to maintain herd immunity.
Measles is one of the most contagious viruses known to man. It has a R0 of 18, meaning a herd immunity threshold of ~95%. Fortunately the Measles vaccine is one of the most effective vaccines we have available. The full two dose series has a vaccine effectiveness of 97% against infection. As a result of such a powerful tool we managed to eliminate Measles from the Americas in 2002.
We've had a few recent scares since then with traveler sourced outbreaks in low vaccination communities.
When we account for medical exemptions we are left for little additional margin for other exemptions. Especially since herd immunity is a local phenomenon and philosophical and religious exemptions aren't necessarily uniformly distributed. It is quite common to see religious communities fall well below the recommended Measles vaccination rates via religious exemptions.
Fortunately the coronavirus is not nearly as contagious as Measles, but the vaccine is proving less effective against infection and transmission. Herd immunity might be elusive with the Delta variant. But even still the vaccine reduces infection and transmission all else equal such that we should strive for very high vaccination coverage to protect our most vulnerable that aren't able to get the same vaccine protection as others.