It is probably comparing 2 different things, there maybe 3 million single basepair differences between you and me.

And 30.000 gene differences between us and neanderthalers, but a gene that is different, can be a single basepair that is different, or a hundrend different basepairs.

I bet if you look at single base pair differences between us and neanderthalers it will greatly outnumber the differences between you and me.

Take this next part with a grain of salt. But i do remember the genetic similarity between you and me is 99.99%, and with us and neanderthalers our dna is 99.8% similar.