You have people that had their brain compressed to half its size by fluids and they were perfectly fine. Intelligence is more about connectivity between neurons, epigenetics and ability to solve problems in current situations than absolute number of neurons.