I agree with you, that's exactly what I mean when I say that it's missing the point.

Even when he's being "racially insensitive", he's not doing it to promote a bigoted ideology or as an easy way to fame like modern right wing comedians are doing, he's doing it for the exact opposite.

It's like being mad at Schindler for making Jews work in his factory, basically.