Socialism proper refers to the advocacy of some sort of collective, social, or public ownership of industry/enterprise and, usually, of land as well. We find many different varieties of socialism. Ricardian socialists advocate replacing private ownership of the means of production with worker-owned cooperatives. Marxists advocate public ownership by the State, which in turn would be run by the proletariat. Lange-Lerner socialism advocates public ownership of enterprise within a market system with central planning. Then there are the libertarian socialists, like Proudhon and Bakunin, who advocated worker-owned cooperatives alongside public ownership of land at the municipal level. Kropotkin’s variation of libertarian socialism had communal ownership, like decentralized Marxism minus the State, rather than independent worker-owned cooperatives as with Proudhon's mutualism and Bakunin’s collectivism. But all varieties of socialism have one common characteristic: they oppose private ownership of the means of production in favor of some sort of social ownership.
Capitalism, unlike socialism, refers more to the condition of things than the precise form of ownership. Capitalism is not private ownership and markets. There are countless examples of market economies with private enterprise that we would not regard as capitalism. Feudalism is a key example, as was medieval distributism. An essential characteristic of capitalism is the division of society into two classes, haves and have-nots, where the have-nots must work for the haves as wage-laborers in order to survive. Under capitalism, most people do not have any share of private ownership over the means of production, meaning that a small group of capitalists and landlords essentially rule over the masses. The vast majority of people under capitalism are not capitalists, which means that they must sell their labor to the capitalist class in order to survive. Wage-slavery is an essential characteristic of capitalism. For a system to be capitalism, it must have money, markets, and private ownership. It must be industrialized. Those are necessary but not sufficient conditions for classifying a system as capitalism. To be capitalism, wage labor must also be the normal mode of survival. Markets, private ownership of land and enterprise, and wage-slavery are the three necessary conditions that must be met in order for a system to constitute capitalism. There are other systems that also have division of people into haves and have-nots but do not constitute capitalism. Feudalism may have markets, private property, and division into haves and have-nots, but it doesn't have wage-labor as the norm and does not have an industrialized economy, so it is not capitalism.
To some extent, we are making a semantic choice in choosing these definitions. We are, of course, excluding some people that would have regarded themselves as “socialists.” For our purposes, we will say that any system with private ownership of industry and enterprise is not socialist. This means that Benjamin Tucker and his individualist anarchism is not actually socialist as far as we are concerned, even though Tucker and his followers self-identified as libertarian socialist. Additionally, we must say that social democrats like Eduard Bernstein, Anthony Crosland, and Anthony Giddens, are not socialists as far as we are concerned. They still advocate a system with private ownership of industry and enterprise. They do not advocate social ownership of any sort.
Tuckerite individualist anarchism is rooted in Ricardian socialism and mutualism, which is why Tucker called himself a socialist. Nevertheless, Tucker departed from those socialist traditions insofar as he did not advocate any sort of communal, collective, social, or public ownership of land and industry. He accepted the socialist critique of capitalism and accepted certain ideas that were proposed by the socialists (e.g. rejection of fee-simple in favor of usufruct per Ricardian socialism, advocacy of mutual banking per Proudhon, etc), so it made sense for Tucker to identify with the socialist cause. We, however, see his sympathies with socialism to be with the “accidental” (i.e. non-essential) parts of socialism rather than with the “essence” of socialism.
The social democrats have a similar story. Social democracy originated in Marxism and state-socialism. First, there was Fabian socialism, which thought that socialism could be brought about through gradual democratic reforms. They rejected the Marxist idea that violent revolution was necessary. They noticed that many industries tend towards monopoly under capitalism and that monopolies tend to be taken over by the State in order to serve the best interests of the people. They thought that socialism was an inevitable result of the progress of capitalism. For example, municipal water provision and provision of electricity tend to turn monopolistic and municipalities tend to take over such industries in order to keep monopoly prices from ruining a good thing. They thought democratic reform could speed this process along. Over time, it became obvious that actual socialization of industry doesn't always occur. Sometimes the government just regulates the monopoly and sets prices for it in order to prevent the negatives that come with monopolies on the free market. It also began to look as if exploitation and the immiseration of the proletariat could be counteracted by regulation. Things like OSHA, the EPA, and minimum wage laws helped to move capitalism in the direction of something somewhat analogous to socialism. Social democracy today is not a perfect analogue to the socialist ideal, but it will get there eventually.
Tuckerite individualist anarchism and social democracy are anti-capitalist, largely following the socialist critique of capitalism, but they are not forms of socialism proper. They don't advocate any form of social, collective, or public ownership. Instead, these philosophies fall more into the “third way” category, alongside distributism and geoism. “Third way” philosophies advocate something that is neither capitalism nor socialism. Advocates of third way political economy seek the abolition of wage-slavery but do not regard public ownership of industry as the appropriate solution. That's essentially where I stand.
The definitions I have chosen are somewhat arbitrary, I will admit, but they are useful and accurate. I believe these definitions are more accurate than any alternative definitions. The definitions of “capitalism” and “socialism” that I have chosen are, firstly, consistent with the way those terms were used by the people that coined the terms and, secondly, the most widely applicable definitions that retain a distinct meaning. If I broaden the terms any wider, the lines between the two become so blurred that the terms lose meaning. If I make the definition any narrower or broader, then it also becomes no longer consistent with the way that those terms were originally used. The whole purpose of language is accurate classification and I don't think that any other definition of either term is as accurate when overlaid upon reality.