Human emotion is seldom logical, even if one can find an evolutionary reason for their existence. They are only rules of thumb than on an average have increased fitness in a world much different from our own. What is well known is that most of us are born with a gender identity that long-term is emotionally very costly for us to go against. (See the result of tragic surgical experiments on intersex children.)

Our brain is optimized to find patterns. In a society were only women wear skirts, an individual with a completely masculine identity will not want to wear one. Of course a skirt has nothing innate feminine about it, but in that culture it does.

In a hypothetical society were the only difference between men and women was our bodies, gender dysphoria would still exist. But it would probably be more focused on physical differences.

My pet theory is that gender identity is primed by primitive signals, e.g. pheromones (e.g. https://academic.oup.com/cercor/article/18/8/1900/285954?login=false for corroborating experimental findings). During our childhood our neocortex couples that to the patterns in the world we find. That pattern can change depending on society and circumstances.