Remove it all and re-caulk or you'll be chasing the issue for years to come.

The one trick that contractors do is to fill the tub before caulking around the tub.

The weight drops the tub. Otherwise, the caulk will separate from the tub later in when you fill the tub for a soak.