You're right, fuck that.

My advice: pick out something that you're better than your bully. Maybe you're smarter. Maybe you're taller. Maybe you're a better musician. Maybe you're just a nicer person. Who knows? All I know is that there's definitely something you do better than him.

Now use that thing to gain confidence. You're letting your bully decide what's good (i.e. strength). Flip it around and focus on the things that you're good at. If he starts bullying you about the gym, just use some version of "yeah, I'd like to get stronger but it's not really my main focus. I'm focused on _____"

I get the sense you're still in high school, so I know this is easier said than done. But learning that everyone has things they're great at -- and things they're bad at -- is an important part of life and being your own person.