it's because SSNs were simply supposed to be ID tax numbers that you could give to anyone. Someone knowing your tax number wasn't meant to confirm your identity. It was just supposed to track how much you put into Social Security. The IRS tried to prevent people from using it as an ID security measure but eventually just gave up.

It's like how business EIN operates. It's literally just a number to track the taxes that the business is responsible for. In some cases it even replaces the business owner's SSN on some forms.