About half my friends are Yale affiliated, half are not. From the Yale crew, there are certainly those who I wouldn't be surprised if I was the only one they know from outside of the University.

It's an international institution, there are tons of folks who aren't very socially skilled in Academia, there are lots of reasons why people stay within the university socially. There is also no barrier to meeting new people in town. Going to a social event and having someone say "oh, I work at Yale ____" is incredibly common, plenty of folks do it.

On the other hand, depending on what program you're going for you'll have to see if socialization is in the cards whatsoever =p.