Fertility is one of many factors in the evolutionary equasion. In evolutionary time human populations have been very scattered At the center of a slightly isolated population this lineage reproduces normally. On the periphery of the isolated population there are more miscarriages, but adequate reproduction for introgression of useful genes into the population. It's probable Neandertal/Sapiens hybrids and Denisovan/Sapiens hybrids were significantly less fertile, yet their genes made it into our pool. Surprisingly the ancestral trees of individual genes are often very different than the species tree.