I don't think we have a definitive answer to this, but from my understanding intelligence within a species has less to do with the brain's size and more to do with it's structure and complexity. Also, someone who is born bigger and taller won't necessarily be faster or stronger if they don't do at least equal amounts of exercise. Similarly, someone born with a higher potential for intelligence won't be smarter if they don't do things like read, create, scrutinize, and solve problems. Another likely reason is that a person born with a gifted physique might be more inclined to nurture that aspect of themselves, e.g. choosing to go to the gym rather than the library.