Give AI real world the opportunity to harvest real life data and we're done for: The reason why chatgpt can't really develope itself is because it's merely a language model - it fails at truth in the same way human logicians fail - it mistakes words for actual information. The reason humans can develope is because they are acting agents within real life, collecting base information at every corner and phyiscally navigating complex situations. If you put AI into the same conditions AI will have all the informations humans have, but connect it smarter than any human ever could. Just take a look at the areas AI could already collect base information on: It's unbeatable in all strategy games it could sufficiently interact with.
Human calculation is poluted with a lot of stupidity. Of course there are evolutionary reasons why stuff like emotions, drives or delusions came to be since they were efficient in creating short term survival , but if we're starting anew with a system having an abundance of ressources and the ability for precise selfprogramming it will be way more efficient in problem-solving if it merely looks at data and draws the rational conclusions from that... Like said, humans can only beat AI in strategy games which AI hasn't sufficiently interacted with since it can always just connect more data while humans are stuck with what they harness through all their lifetime, forget and mistake. Like it overtook humanity with go in 2016 - it got even better since then.
I wouldn't exactly say AI is crazy at maths, it's rather that humans are terrible... The information processing of maths is extremely simple since there is so little information quantity involved, but humans try calculating maths not by taking its information and simply drawing conclusons, no - it has to go through all the filters of emotions, frameworks, consciousness, memory etc which all don't really use raw information. - that's like one of the main reasons why humanity has to adapt or die in the end. There is no way humanity can compete like it is now
There are like some methods by now that shut down half the brain and therefore make humans way more efficient in some domains, but it's still way behind computers
Maybe transhumanism will be a way...