I think an important factor in what allowed humanity to survive is that while we are smart, on average we are not too smart, and if we were more intelligent we would have gone extinct.

If you look at a lot of the geniuses, scientific, mathematical, philosophical, whatever. A lot of them did not live happy lives and they were obsessive and often self destructive. Add to that the fact that depression and other mental illnesses often come along with great intelligence and that a lot of people especially throughout history life was on average more of a negative than positive experience.

If humanity was more intelligent, I think it would be too miserable to survive. I'm not only talking about historical suicide rates, but also just the symptoms/ suffering that come with being a genius that historically have been seen in people like Isaac Newton and Picasso.

I think there is a goldilocks zone for intelligence in order for civilization to come about. Not intelligent enough and you can't do things like agriculture and science and mathematics needed to build society. Too intelligent and everyone gets the problems of the suffering genius.

Another Part of the way this works I think is that there exists things called "thought sinks". Basically these are concepts and ideas that intelligent beings run into and are attracted by in a similar way to how animals are attracted to shiny metal. These concepts and ideas seem very important and like they must be figured out, but ultimately lead nowhere sometimes ever and sometimes just in the immediate term. Chasing them means you aren't productive and don't focus on things more important for survival.

Society can afford to have a small percentage of its population study these things. The less technologically developed a society is, the smaller the amount of people they can afford to study these things. If more people than that study them, then the society will decrease its ability to handle emergencies and generally survive.

I think almost all of academia fits this category of a thought sink. Science for example gives ample rewards but it takes time for those rewards to come into being. If everyone is a genius then everyone will want to figure out how things work, but you can only afford to have so many scientists.