Well same reason why we have abandoned industrial complexes, old abandoned buildings and such. People need to be willing to do something with that ground. Removing ruins takes time and money too... the amount of just burned down or abandoned buildings that litter our modern landscape is quite vast too.

In many cases ruins also just just stay landmarks

This is today. Back then all they had were hand tools...not explosives, heavy machinery, cranes and all that.

You are not just "removing" structures that are as solid as lets say the roman colluseum

Often times ruins also tend to be mostly foundations.. no one bothered to remove foundations

So they very much did remove a bunch of it. Just.. put dirt over it and use the square as a market or something.

Other times.. well cities get abandoned by conquest too... whoever conquered the place doesnt necissarily see maintainance of some places as needed.

If europe wasn't rebuilt with massive efforts after ww2 most of these ruins would still be here.. this is how many war torn countries will remain to look like for loong times.

Some of these ruins stood for over a few hundred years. They were often large artistic buildings for different cultural things. So people just left it there because it has just always been there. Space and land weren't as tight and it would be more work to clear it away to rebuild then to just build somewhere else.

The ruins that we see today often didn't stand there as ruins for 2000 years. For example, the most iconic ruin of Athens, the Parthenon temple, continued to be used, first as a Christian church, then as a mosque, for centuries (making some small alterations to it like adding a minaret). No need to tear down a perfectly good building if you can still use it.

It only became a ruin after being damaged in a war, and for the next 150 years, the Ottomans couldn't be bothered to either rebuild it nor to tear it down. Both would be a lot of work, and they didn't need the space that bad, so they just used the rubble to build a smaller mosque inside the ruins and call it a day.

Then, Greece became an independent nation, and the ruins of Ancient Greece were of huge importance to their national identity, so they removed the mosque and everything else that had been added after antiquity, and made an effort to preserve what was left of the ancient temple. What we see today is the result of that effort.

Similar stories exist for nearly all ancient ruins in European cities - they continued to be used and were integrated into newer buildings for centuries, and then beginning in the 19th century people started to remove the newer buildings, uncovering what was left of the ancient buildings, and preserving these leftovers (or sometimes attempting to reconstruct them).

New rulers, new religions, new building styles. Why do we see ruins in even modern urban cities? Why are there abandoned train stations in Detroit or Catholic Churches in Gary? The building is no longer useful (what’s the need for a temple to Zeus after adoption of Christianity?), it’s easier to build on a new location than do demolition and removal before building.

Because the ones that got knocked down and replaced aren't there anymore to wonder about. Survivorship bias. Likewise, the ones that got buried survived, the ones that weren't buried, people carted off parts of them to use to build other things.

Cities like Athens, Rome or Constantinople had huge populations in ancient times, but they dwindle with time. According to some estimates, Constantinople had up to 500,000 inhabitants during the 8th century, and shrank to 45,000 at the time of the Fall. Rome had 800,000 people in 400 AD and 30,0000 in the middle of the 6th century. Athens had up to 600,000 in the 5th century BC, and by 1833 it was a town of only 4,000.

A big city need people to take care of it. When their populations shrank, the people try to stay close together, so entire neigborhoods in the outskirts had barely any people. And with barely any people, maintaining a building, or knocking it down to build something else, is not feasible in the majority of cases, because there's no need to use the place as a building.