You're somewhat right about labor costs. However there is an absurd amount of specialized labor needed to make the fabs run at the cutting edge. Look at how Intel tried and failed falling behind TSMC.

The other issue from a corporate perspective is the fabs are silly expensive. I'm pretty sure they are the most expensive buildings in the world once you account for all the equipment you need . Once you construct one, you need it to run 24/7 pumping out chips to even pretend to make the expenditure worth it. Hasn't the USA had some large profile power failures in the not to distant past? You'd be screaming at the moon as a fab if you suffered a shutdown caused by inadequate infrastructure outside your plant.

China is also trying to get at the high end of chips, and they earmarked over $140 billion USD, and remember that with purchasing power parity that money goes a lot further in China vs USA. All this is to say it is very expensive and technically difficult to be at the high end of fabs. I have a feeling both countries will end up with some higher end fabs, but give it time and they'll fail at keeping up the pace.