The biggest problem with this is that we don't have an affirmative right to vote. What we do have is a bunch of negative rights, mostly the right to not be discriminated against for x, y, and z for the purpose of voting. We do this because it is still relatively popular to discriminate based on other reasons (eg not allowing a person to vote while in prison for a felony). So people can use the fact that we don't have this affirmative right to vote to chip away at it to get much of the same effect as other forms of voter discrimination.