If I maliciously break your leg, what I owe you in recompense is not reduced by the fact that I could have made greater profit for myself by using that time and effort to, say, design a new mousetrap instead of assaulting you. The fact that we are both poorer because of it does not make us both equally the victim.
There are plenty of good and correct arguments against the feasibility and justice of modern-day cash reparations for slavery, including that they wouldn't actually work to their intended goal, that the amounts owed are hopelessly incalculable, that the costs of getting the state into that business would far outweigh the benefits, and that it's unjust to hold anybody alive today accountable for the sins of their ancestors. I don't endorse any kind of race-based government payout, and neither do any other libertarians I know of. I also don't think it's a terribly likely policy outcome that will ever be seriously entertained, so I find the claimed urgency of rebutting it a bit overwrought.
Granted, the modern racial wealth gap is not because of the hoarded stolen profits of antebellum slave labor. There's no piggy bank full of cotton plantation profits to go raid and redistribute. That tells us nothing about what, if anything, is owed to the victims of slavery and Jim Crow (the latter most likely the greater of the two if we're talking about persistent effects on those alive today). It only tells us that present-day costs of the injustice are more in the form of opportunities denied and potential wealth never created, than in the form of wealth created and then stolen.
Regardless of how you figure the GDP% contribution of slavery circa 1860, there are today millions of Americans who are systematically poorer because of centuries of state-sanctioned and state-enforced racial oppression. When people make the earnest case that this is a problem of some moral urgency and which should be accounted for in our public policy debates-- first and foremost by not continuing to compound it-- replying that slavery was a net drag on the antebellum economy is both true and trivially irrelevant.