Your mentality needs to change. You wouldn't be an asshole to not care about people who aren't worth caring about. If someone abuses you have every right to cut off feeling empathy for that person. Easier said than done but when you cut off a person completely time makes it easier to not care. Find people who are worthy of your empathy instead of wasting on those who don't.