Species is just an arbitrary classification. Interbreeding is only one factor used to determine what is a species. Its thought that only female neanderthal human hybrids were fertile and not males, so that one justification for considering us seperate species. Just how likely an offspring is to be fertile could also be taken into account. If two species have to breed a million times to produce 1 fertile offspring, it doesnt mean the two are the same species, there is never going to be considerable gene flow between those two groups.