Blame Beau Brummell.

He was the guy who, in the early 19th century (so during the Napoleonic Wars) started and popularized the male standard of highly tailored and neutral coloured or monochromatic black and white clothing in luxury fabrics. Brummell was good friends with the Prince of Wales, who was then the Prince Regent b/c King George was mad, and of course Brummell’s taste in fashion as extolled by the guy who ruled Britain meant that men of means were quick to follow suit (pun not intended).

Before then Western men’s fashions were as colourful and excessive as the women’s, and for the same purpose - to show off one’s wealth and taste. Compare a Tudor to a Georgian nobleman and you’ll see similar use of colours. But Georgian-era dandy to a Victorian gentleman? And that was in only 50 years.