Capitalism
Capitalism has been defined by its critics as a system in which a small group of people own the means of production, while the majority of the populace, lacking ownership of productive property, have no choice but to sell themselves to the owners of said property as wage workers. A system in which this is the case must necessarily have the following features: private-ownership of the means of production, a monetary system, and a market. These features are necessary but not sufficient conditions. If a system lacks these three features, it is not capitalism. If it has these features, it may or may not be capitalism.
The thing to note here is that capitalism, so defined, is not so much a system as a condition. Capitalism is not a market system, but rather a market system in which economic power is concentrated into the hands of the few and the majority of the populace is dependent upon, or subjugated to, the owners of land, resources, and machines used for production. Thus, there can be anti-capitalist market systems: e.g. Ricardian socialism, Tuckerite anarchism, distributism, mutualism, Fabian socialism, and some varieties of social democracy.
Socialism
Socialism refers to any of a variety of economic systems (and corresponding ideologies) that seeks to eliminate capitalism through the abolition of capitalistic property. Usually, socialism seeks to abolish the institution of private property over the means of production in favor of some sort of public or collective ownership. For instance, Proudhon's libertarian socialism advocated municipal-ownership of land and worker-ownership of industry, Marx’s vision of socialism entailed government-ownership of industry, and the Fabian socialists envisioned a mixture of municipal-ownership and national-ownership of land and industry. Some variety of public or collective property was the typical vision of socialists.
Third Ways
Beyond capitalism and socialism are several “third way” philosophies that can be seen, or have described themselves, as a third way beyond the capitalism-socialism dichotomy.
There are some who have called themselves “socialists” who depart from the standard socialist vision. Tuckerite libertarian socialism and Ricardian socialism advocated replacing fee-simple property with usufructuary property, holding that ownership ought to be linked to occupancy and use. Then there are some social democrats who refer to themselves as “socialists,” but really hold the Nordic Model of social democracy up as their ideal. They generally advocate universal healthcare, social security, and other universal welfare measures, and see private-ownership of industry as no problem at all. These, perhaps, are better labeled pseudo-socialists.
Capitalism and socialism are not the only options available. Alternatively, there is distributism. Distributism, like capitalism, does not designate a system per se, but rather refers to a condition. Distributism is widespread ownership of productive property, such that wage-slavery is eliminated. In a distributist economy, most people either own their own business or have a share of ownership in some cooperative enterprise. Such an economy must have private property, a monetary system, and markets, without meeting all the criteria necessary for constituting capitalism.
There is also Georgism, which entails using taxation as an analogue to public-ownership of land and natural resources. This has sometimes been called a mixture of socialism and capitalism, but perhaps is better understood as a third way or alternative to both. Georgists advocate free trade. Since Georgism does not fit either of the two conventional labels, it has serious proponents on both sides. Georgism has been advocated by prominent proponents of capitalism, such as Milton Friedman, as well as by Fabian socialists and mutualists. Georgism is compatible with these ideologies but coterminous with none of them.
Some advocates of Nordic Model social democracy have come to recognize that modern social democracy is no longer socialist. These social democrats have opted to refer to social democracy as a “third way” rather than as being either capitalism or socialism. Their status as “third way” is somewhat dubious, since the economies of Nordic countries still meet all the criteria for qualifying as capitalism. The curious thing about Nordic Model social democracy is that, although historically related to democratic socialism, today it has less to do with socialism and more to do with the ideas espoused by Franklin Delano Roosevelt in his Second Bill of Rights.
Additionally, there is an alternative model which centers on the idea of combining Georgist land value tax with a universal basic income and also providing the same universal welfare programs typically associated with Nordic Model social democracy. Personally, I have referred to this as “libertarian social democracy.” There are others who have advocated this without using any such label. This position is mostly rooted in civic republicanism, being inspired by Thomas Paine, Thomas Jefferson, and other classical republican theorists.