The dirt is just a collection of debris over the years.
It's certainly more common with dirt roads outside, rather than it being paved and somewhere to clean back to.
Here's a good vid showing doors raised over time as the street level rises
https://www.youtube.com/watch?v=fz4ZdXpri04