The Egyptians didn't have chariots or horses until 1600 BC so their main method of supply/troop transport was via boat on the Nile. Wherever the Land of Punt was, it didn't seem to be reachable by the Nile.

If Punt was in Arabia, the Egyptians would have had to build an expensive invasion fleet of seaworthy ships, which is outside of their expertise with river ships. If Punt is in Eritrea or Ethiopia, it's too far from the Nile; there is a tributary that flows in that general area called the Atbarah AKA Black Nile but it's not as big or deep as the Nile and not conducive to large boats.

By the time the Egyptians got horses, Kush/Nubia was very strong and the Egyptians had too many problems in the North (the Hittites, whoever was in control of Mesopotamia, the Sea Peoples) to really expand into where Punt might be.