When only a select few vote, it’s safe (and indeed effective) to completely ignore the wants/needs of the non- voters.
This leads to extremism- the easiest way to get more for your chosen few is to take more from the groups you don’t care about. This creates a feedback loop where you’re now incentivised to make it harder for those groups to vote (since you know they already dislike you) which means fewer of them vote and they become more ignorable and more ripe for you to squeeze in favor of your chosen. It also means you can focus for support from a group that represents a smaller and smaller percentage of the populace, because all that actually matters is what percentage of the voters they are.
Alienating the more moderate parts of your original base doesn’t matter - they’ll just drop off the rolls (or be cleared off by you) hell your vote percent might actually improve.

When the populace and the voters are always the same demographic this kind of thing can’t work.
It’s pretty straightforward math.

Extremism profits from making the demographics of voters heavily skewed from the demographics of the populace. Why would you allow that to happen when it’s simple to prevent with mandatory voting?