Fucking punch back. One of two things happen. One, they acquiesce and y'all become even better friends, albeit ones who can still rip on each other. Two, they double down, and you realize (unfortunately) that you fucked up because they were never your friends. Which leaves you with fewer friends, sure, but thinking how much life you have left and focus that on surrounding yourself with the right people.

Worried you'll spend a lot of time alone? Face that shit head on and based on your personality (introvert, extrovert), find creative ways to meet new people. Or actually face it and realize that being alone for periods of time is a hallmark of human existence; it doesn't make you special, weak - it makes you boringly "normal."

Good luck OP.