Essentially, the rate of mutation in individuals may remain constant, but without some environnemental pressure exerting a selective effect on individuals with advantageous mutations, there is no reason for those mutations to become reinforced each generation.

If 1/1,000 people randomly have slightly longer fingers, but there is no pressure in the environnement that grants long-fingered people an advantage reproducing, you won’t see an increase in long-fingered people over time; there will remain a roughly 1/1,000 chance that anyone you meet is long-fingered.