MD here. I believe the answer is unless you fall under the EMTALA rules/need emergency care a doctor can legally refuse to care for you for a variety of reasons and refusal to sign the release would be a good one. PHI release is necessary for appropriate handling of health information in an organization, quality assurance, and for billing reasons.

Caveat: I’m not lawyer and personally haven’t had a patient refuse to sign the release ever