Imagine getting hit with a baseball vs a ball of paper. Same speed at you. It comes down to weight.

When someone punches, swings a bat, ect, they technically are throwing a percent of their weight behind the action. More weight =harder

Harder punch = weight x speed (ELI5 version) Force= mass x acceleration