I think it's perfectly reasonable to abstain from forming a belief, but I think there's plenty of situations in which it's reasonable to form beliefs even in absence of proof.

This is the case in many ordinary situations. Suppose I meet a couple and they tell me they're married. They wear wedding rings and act like a couple. I can't prove that they're married, but I have a substantial amount of evidence suggesting it's the case and no counter evidence. There are plenty of scenarios I could concoct which could be unprovable, such as that they're foreign spies or visiting aliens with a sham marriage as part of their cover story.

I don't encounter these scenarios and abstain from drawing conclusions on the basis of their unprovability: I construct beliefs on the basis of a preponderance of evidence. Colloquially, I might even say "I know they're married", even if I can't prove true belief.

I think a major difference between math and everyday epistemology is that the vast majority of math I encounter is provable, while the vast majority of everyday "knowledge" is premised on things that are not.