The Colosseum has been under control of many different people and used for many different purposes. Each change resulted in modifications (ie the wooden seats were all removed to be used as barricades when people started using it as a fortress instead of a stadium).

A lot of the building has fallen down due to earth quakes. Some of which are quite recent. There's markers around the outside that show where the REAL outer walls should be. It used to be a lot bigger.