There's a lot of truth to what you're saying, but there's an inherent natural inclination to see up/down as fixed and side/side as variable (seeing as that's where we do and don't have total freedom of movement) and using the (more or less) fixed poles as the fixed point makes a sort of sense that would arise naturally, I think.