Even that's understating it a bit. You could put 5 other objects in Earth's orbit around the sun, as far from their neighbors (including Earth) as they are from the sun.

And asteroid sizes roughly follow a power law, with many, many more small asteroids than large ones. The sixth most massive asteroid has only 2% of the mass of the 5 most massive asteroids, which together hold over 60% of the mass of the entire belt. You could put 60% of the belt's mass in Earth's orbit without any of it getting closer to Earth than to any of the other objects or the sun itself.