I think this is a case of "we can't tie those 3 million differences to neanderthals specifically". Maybe they also had them but we don't have enough sampling to know. What we do have is relic neanderthal genes that have mostly spread throughout all the population in the few hundreds of generations since they mixed in.

It is a misleading statistic. The extent of variation in neanderthals is poorly known simply because identification of that type of variation requires thousands upon thousands of samples to be analyzed, and we don't have that. All we have is enough data to say what all neanderthals had in common with each other (what makes them specifically neanderthal). The extent of variation in existing humans is well known because there are millions of analyses. Not really comparing the same details either (comparing apples to oranges, in a way). Major components that are unique to Neanderthals are being compared to major components of existing humans in the one case, and in the other, trace components among humans are being compared to trace components in other humans. They don't differ from other humans in the 10,000 ways (almost?) all humans differ from neanderthals.

Comparing apples and oranges and saying they are different in 10 easy to identify ways ways and then pointing out there are hundreds of varieties of apples, and then pretending that this proves that apples are more varied than oranges.