It wasn’t so much that it was vanity as it was the legacy of games being a pagan religious activity. We forget that the ancient games were first and foremost religious festivals that incorporated ritualized combat (all of the ancient Olympic events were first and foremost combat skills - not that hand-to-hand-combat-ballroom-dancing doesn’t have a certain appeal in theory…).

It is true that as the empire turned away from paganism it eschewed various aspects of civic and private life that were incompatible with the Christian ethical/moral construct. Remember, the festivals were not some sort of neutral, secular, agnostic affairs. They were first and foremost acts of religious devotion and expressions of faith. And since Christianity (and Judaism for that matter) has a pretty strict prohibition about worshipping other gods, it all had to go.

Thus we see the end of gladiators fighting to the death, the large games (Olympian, Delphian, etc.) and a wide range of other activities, events and festivals. The ancient philosophic schools also declined as society morphed and changed; they were replaced by other schools and institutions more in line with the contemporary mores of the period.

As the gymnasiums were a fusion of the pagan philosophical tradition and the pagan religious athletic tradition, it faded as well. I am very interested to learn about how that particular area of interest evolved though. It isn’t as though athletes suddenly disappeared or that fitness and martial prowess went up in smoke. Like mosaicists, sculptors, etc. who used their skills & interests to praise the pagan gods and the Christian God, I am sure athletes found another similar outlet. And I don’t know about it.

There has been some interesting work around the decline of the large public bath. The latest thinking being that the public bath declined primarily as the result of environmental and economic factors. Baths consumed vast amounts of water and fuel (to heat the water). This cost vast amounts of money. Throw in the persistence of ideas around virtue and vice (Christian frowning on men and women being naked/bathing in each others company was not new or novel in any way) and presto, end of the large public baths.