>Sex is a classification because it is part of material, empirical reality

There is no such thing as a classification outside of the abilities of a sense-making being. It doesn't mean that, if humans don't exist then there are no such things as, say, apples. What it means is that a classification is explicitly the work of sense-making beings (e.g. humans).

>That's like saying that because some people have a birth defect that gives them six fingers on each hand, that humans as a species don't have five fingers on each hand.

No, the appropriate analogy would be to question "birth defects". What makes a person with six fingers "defective"? Establishing a norm requires looking at variation and deciding what the norm is for. If the norm just means "statistical average", then that's fine, but it doesn't mean that a person with six fingers doesn't have a hand, it just means that it's a hand differently composed than the statistical average. Intersex classification is differently composed from the statistical average of male/female and so cannot be subsumed as either one.