Great post again, Spectrum! I hope you don't mind if I contribute a few thoughts of my own, all of which are :

There is no doubt that, in the overall aggregate, the benefits of specialization and trade have far outweighed the costs... but this does not mean that there aren't costs to watch out for all the same.

There are, then, three potential downsides of which we must remain weary:

1. Mechanization. OP already introduced the idea that by specializing in the same, overly-repetitive action, we can specialize ourselves into a boring, if not miserable existence.

2. Over-specialization. Specialization is a process in which larger tasks tend to be broken down into smaller ones. The smaller and more specialized the task becomes, however, the more likely is it to be replaced by a machine. As such, there is a threat that I will dedicate my life to specializing in X, just to have my job replaced by a machine, at which point I am no longer qualified to "specialize" in any other paid job.

3. Division of Responsibility. The division of labor necessarily brings with it a division of knowledge and a division of social/moral responsibility. For example, Europeans in the 19th century could easily condemn New World slavery as barbaric while at the same time supporting that institution through their purchasing decisions. Indeed, since low-cost, slave labor had pretty much put all competition out of business in some markets, the European consumers had little option but to support New World slavery. Specialization does not necessarily create these contradictions, but it does make them harder to identify and eradicate.