We don’t necessarily, it depends on the extent and type of bacteria causing the mould, your personal susceptibility to that bacteria, and indeed how your body handles exposure - such as digestion.

It is a bit of a misconception that mould is bad in absolute, but for the ease of removing obvious mould from food - such as cutting the mouldy bit off some generally un mouldy cheese - you may as well do it.