"If we represent the oppressed then it's only fair we represent the oppressors too" is an absurd cop out. As a queer person i don't particularly care whether a a pride flag flies at town hall or not because it's a performative measure that makes no material difference and isn't an indicator of actual representation. But, if they claim the reason they won't fly it is because they're not capable of making informed decisions about any other flag going forward, then they're not competent representatives.

You're completely correct about all flags being political.