I think a lot of it goes back to the fusionist era (pre-Rothbard and pre-paleo-ism) and the correct desire to repudiate and denounce the “egalitarian” foundations of socialism. This eventually morphed into unqualified denunciations of egalitarianism as such, and positing an inherent dichotomy with equality as the antithesis of liberty. Rather than, as properly understood, the premise that is a necessary predicate of liberty. This swept broader than the original socialist notion of material egalitarianism, and eventually hit at the liberal foundations of “all men are created equal” rights egalitarianism.
If you have no philosophical commitment to the liberal egalitarian premise and the conclusions that flow from it (cf. the Declaration of Independence among many others), or even see talk of “equality” as your utmost enemy, then you can’t have a theory of universal human rights. You have a theory of class privileges and an ugly argument over who gets them. By making that leap, they place themselves not only outside of libertarianism, but outside the entire broader liberal tradition that libertarianism is a subset of. You can have authoritarians who happen to want to slash welfare or cut overall government spending or abolish the Department of Education. But if they don’t defend the universality of the rights of all persons, and instead draw categories of persons whose rights we are not bound to respect, then they aren’t any species of libertarianism.
If you look at the 19th century classical liberals and proto-libertarians, their rhetoric is rife with talk like “Equal Rights for All” and the “law of equal freedom” and the “equal protection of the laws.” They understood that claims to inherent human equality and claims to liberty were two sides of the same coin; two ways of stating the same fact about human nature. Rejecting one is rejecting the other, and the most obvious and still widely accepted way to do so is on the basis of nationality, followed closely by race.
That’s basically what we’re dealing with at its core; the conceptual and rhetorical errors of decades past that led to this ideological trainwreck. It’s the shell of a libertarian small-gov’t policy agenda wrapped around an exception that swallows the rule. A bait and switch that starts with proclaiming itself dedicated to a theory of universal rights, and then smuggles back in a division of humanity into who does or does not have those rights. It’s a pipeline from libertarianism to the far-right not just in practical terms, but also conceptually.