Right there with you! Rousseau even addresses the failings of the Social Contract in his own writings:
Rousseau in The Social Contract:
“Even if each man could alienate himself, he could not alienate his children: they are born men and free; their liberty belongs to them, and no one but they has the right to dispose of it. Before they come to years of discretion, the father can, in their name, lay down conditions for their preservation and well-being, but he cannot give them irrevocably and without conditions: such a gift is contrary to the ends of nature, and exceeds the rights of paternity. It would therefore be necessary, in order to legitimize an arbitrary government, that in every generation the people should be in a position to accept or reject it.”