Important question: are you waiting until you're already cold to put on your gloves?

If so, put them on earlier. It's easier to keep warm than to re-warm.