Sure, it can be abused, but it's also easy to overturn someone that's doing something that people don't like. My HOA just swapped out the entire board and president because the residents didn't like the management company we hired because they were too stringent about writing infractions. Took a whole 20 people out of over 800 to vote the new board in.

In my experience, anybody who complains about their HOA is either too lazy to get involved and spend an hour a month to go to the meetings, or they're the kind of neighbor that nobody wants.