It's absolutely a way to distract you from your feelings.

Again, I really don't believe your boyfriend is thinking of anyone in particular when he says "other women don't have an issue with it". It's just an excuse he throws out there to feel more justified in continuing his inappropriate behaviour. It's like when a man is called out for cat-calling and claims "none of the other women I yelled inappropriate things at ever complained to me about it".