No it has to do with the volume. People usually talk extra loud on speakerphone and have the volume up extra loud. It's distracting to the ear versus hearing two people talking next to you.

I also remember reading once that scientists tried to study why we also find people who talk on the phone in public annoying. I remember them determining it had to do with our brain not liking only hearing one side of a conversation. I think it's just because people never talk with the same volume on the phone as in person. They are regulating their volume for someone who is not there and it rarely jives with ambient noise in the space they are in.