To be clear, when people say the Sahara was once "green", don't picture a rainforest, picture something like the serengeti. So there wasn't a ton of organic matter there to start out with.

But anyway, water and wind erosion are constantly working to remove topsoil, and vegetation plays a big role in securing it. Once the climate dried and the vegetation died out, the soil eroded away and the bedrock behan eroding, producing sand--though significant portions of the sahara are just bare bedrock or desert "pavement" which is compacted rock.