I've done this, and I told the person on the phone all about the rude jackass on speakerphone. I made a point to be audible and loquacious but polite and not SUPER insulting. Just making fun of him for bad manners and situational unawareness.

When dude noticed and started to say something like "hey, fuck yo-" I just made a cartoonishly animated show of covering the phone mic, holding it away, and like a rabid weird Al yankovic screamed "IM ON THE PHONE!" At him with the "just a minute" gesture.

I think I outcrazied him, because he did not opt to continue our interaction. He also didn't continue his phone call, but I'm pretty sure it just made everybody else more uncomfortable than if I'd not tried to intervene.